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Abstract 

The  purpose  of  this  research  is  to  investigate  the  effects  of  data  presentation 
methods  on  technician  performance  when  the  procedures  are  presented  on  a  monocular, 
head-mounted  display  (HMD)  in  a  static  maintenance  environment.  This  research  used 
two  different  methods  to  present  the  maintenance  task  data  to  the  technicians.  The  first 
method  showed  the  task  as  it  is  typically  described  in  standard  technical  manuals.  It 
described  the  task  to  perform  and  provided  a  basic  picture  of  the  cannon  plug  to  be  tested 
(unenhanced).  The  second  method  provided  the  same  information  as  the  first,  but  it  also 
modified  the  information  by  providing  visual  cues  as  to  which  pins  were  to  be  selected 
and  connected  (enhanced). 

United  States  Air  Force  avionics  maintenance  technicians  stationed  at  Barksdale 
Air  Force  Base,  Louisiana  were  the  test  participants  in  this  study.  Measurements 
included  task  completion  time,  task  error  rate,  and  technician  self  reports  on  the  HMD 
usability.  The  technicians  indicated  that  HMDs  could  be  a  useful  tool  in  the  performance 
of  their  maintenance  duties.  The  data  collected  during  this  study  indicates  that  the 
technicians  performed  the  tasks  quicker  and  committed  fewer  errors  when  they  used  the 
enhanced  graphical  data  presentation  method  to  perform  the  experimental  tasks. 
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REDUCING  CANNON  PLUG  CONNECTOR  PIN  SELECTION 


TIME  AND  ERRORS  THROUGH  ENHANCED  DATA 
PRESENTATION  METHODS 


7.  Introduction 


Background 

Since  the  early  1980s  the  Department  of  Defense  (DoD)  and  commercial  industry 
have  been  moving  to  digitize,  computerize,  and  automate  the  technical  manuals  that  are 
used  by  maintenance  technicians  to  troubleshoot,  repair,  inspect,  service,  and  maintain 
weapon  systems.  Automated  maintenance  aids  —  maintenance  aids  that  are  based  on 
computer  technology,  and  integrated  maintenance  aids  —  that  provide  interactive  system 
tests  and  access  to  required  technical  information  for  troubleshooting  and  repair,  are 
beginning  to  be  fielded  more  frequently. 

Maintenance  is  continually  performed  in  less  than  ideal  conditions.  Technicians 
are  often  working  out  in  the  hot/cold,  wet/snowy  weather,  within  confined  places,  dealing 
with  limited  part  accessibility,  and  needing  both  hands  to  perform  a  task.  Any  automated 
or  integrated  maintenance  aid  must  be  able  to  perform  in  these  conditions  in  order  for  it  to 
be  a  true  asset  to  the  technician.  Maintainability  and  integrated  maintenance  are  being 
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incorporated  into  new  weapon  systems  procurements.  Many  systems  currently  fielded 
are  being  investigated  to  see  if  they  can  be  retrofitted  with  more  automated  maintenance 
aids  (Kolleck,  Booz,  Allen,  &  Hamilton,  1994). 

Research  has  shown  that  automated  maintenance  aids  improve  maintenance 
technician  troubleshooting  performance  —  not  only  in  problem  identification,  but  also  in 
task  completion  times  and  in  reduced  maintenance  costs  (Schroeder,  Smith,  Bursch,  & 
Meisner,  1992;  Thomas,  1995).  Knowing  that  automation  and  computerization  can 
improve  performance  is  one  matter,  but  getting  it  out  on  the  flightline  or  any  other  place 
where  the  maintenance  is  being  performed,  is  quite  another. 

Armstrong  Laboratory  at  Wright-Patterson  AFB,  Ohio  has  been  conducting 
research  to  develop  and  evaluate  technology  for  an  Integrated  Maintenance  Information 
System  (IMIS)  since  the  1970s  (Thomas,  1995).  Results  from  the  IMIS  test  and 
evaluation  process  has  provided  the  promise  of  reducing  maintenance  technician 
troubleshooting  errors  through  the  use  of  computerized  maintenance  data  presentation 
methods  (Thomas,  1995).  These  presentations  are  computerized  versions  of  the 
technical  manuals,  troubleshooting  trees,  and  parts  diagrams  used  by  technicians. 

Aircraft  maintenance  technicians  were  tested  to  see  if  computerized  data 
presentation  methods  would  improve  their  performance  by  reducing  the  time  to  complete 
specific  tasks  or  reducing  error  rates  (completing  the  wrong  task  or  making  the  wrong 
troubleshooting  decision).  The  tests  showed  that  the  technician  error  rate  was  reduced 
using  the  computerized  data  presentation  method.  The  largest  source  of  observed  error 
came  from  the  improper  use  of  test  equipment,  with  selecting  the  wrong  cannon  plug  pin 


2 


being  the  largest  contributor  (Thomas,  1 995).  Improper  cannon  plug  pin  identification 
can  be  a  serious  problem,  misapplication  of  test  leads  can  cause  thousands  of  dollars  of 
damage  by  improperly  routing  voltage  to  the  wrong  circuit  (Shepherd,  1990). 

Reducing  technician  error  rates  in  this  area  can  thus  save  time  in  troubleshooting 
and  money  in  replacement  parts.  In  this  era  of  reduced  budgets  and  reductions  in  force, 
any  affordable  technological  advantage  that  can  be  utilized  to  improve  the  performance  of 
maintenance  technicians  should  be  pursued.  Improving  performance  reduces  the 
troubleshooting  time  required  to  return  equipment  (aircraft,  ships,  vehicles,  etc.)  back  to 
fully  mission  capable  status.  Quicker  turnaround  means  that  the  equipment  is  capable  of 
performing  its  designed  combat  function  more  often. 

Automation  and  integration  can  also  be  used  to  train  technicians.  Computers  can 
be  used  to  simulate  a  real  system  in  the  training  environment.  Training  technicians  on 
simulators  allows  them  to  make  mistakes  and  learn  without  damaging  the  real  system. 
Thereby  saving  money  in  procurement  of  training  assets  and  repair  costs,  while  still 
providing  the  needed  training  for  the  maintenance  personnel. 

Automated  and  integrated  training,  troubleshooting,  and  repair  aids  require 
another  important  ingredient  —  the  ability  to  be  used  in  the  “real  world.”  That  is,  they 
need  to  perform  where  technicians  work,  and  they  must  function  as  they  do  in  the  sterile 
training  environment.  In  the  Air  Force’s  case,  one  example  would  be  the  flightline. 
Flightline  aircraft  maintenance  technicians  must  be  able  to  use  their  equipment  in  any 
situation,  under  any  condition. 
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Portable  head-mounted  displays  (HMDs)  have  been  studied  to  see  if  they  can 
provide  the  portability,  reliability,  flexibility,  and  durability  required  for  flightline 
maintenance  operations  (Masquelier,  1991;  Friend  and  Grinstead,  1992).  Figure  1  depicts 
technicians  wearing  an  HMD  while  performing  maintenance  tasks.  Friend  and  Grinstead 
(1992)  did  a  comparative  study  between  a  portable  flat  screen  laptop  computer  and  an 
HMD.  The  conclusions  of  their  study  indicate  that  technicians  performing  tasks  with 
data  displayed  on  the  HMD  generally  performed  better  than  their  counterparts  using  the 
flat  screen  (Friend  and  Grinstead,  1992). 


Figure  1:  Maintenance  Technicians  Wearing  an  HMD 


The  present  research  expanded  on  Friend  and  Grinstead’s  (1992)  study  and 
Armstrong  Laboratory’s  cannon  plug  pin  selection  error  problem  identification  during  the 
IMIS  field  test  by  investigating  the  use  of  HMD  technology  in  reducing  cannon  plug  pin 
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selection  errors  through  the  use  of  enhanced  graphical  data  presentation  methods. 
Enhanced  graphics  intends  to  improve  the  standard  presentation  and  consists  of  highlights 
and  visual  cues  as  to  the  cannon  plug  pins  in  question.  Figure  2  contains  examples  of 
both  the  unenhanced  and  enhanced  presentation  methods  for  the  13  pin  cannon  plug  used 
in  this  thesis. 


Figure  2:  Unenhanced  and  Enhanced  HMD  Presentation  Methods 
for  the  13  Pin  Cannon  Plug 


An  HMD  was  used  to  present  two  different  graphical  representations  for  the  same 
cannon  plug  to  technicians  to  see  if  enhanced  graphics  could  be  used  to  help  technicians 
properly  identify  pins  on  the  cannon  plug  and  reduce  pin  selection  errors  made  by 
technicians.  The  number  of  caimon  plug  pins  was  also  investigated  to  see  if  the  number 
of  pins  would  effect  technician  performance. 
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Thesis  Statement 


HMDs  are  currently  used  to  provide  automated  technical  data  to  maintenance 
technicians.  Much  of  this  data  is  simply  digitized  representations  of  the  paper  products 
currently  being  used.  The  paper  data  is  scanned  and  imported  into  computerized  versions 
of  the  original  technical  data,  which  enables  large  amounts  of  data  to  be  readily  available 
to  the  maintenance  technicians. 

Research  has  shown  that  the  computer  version  of  the  technical  data  can  reduce 
errors  made  by  maintenance  technicians;  however,  errors  are  still  made  in  selecting 
cannon  plug  pins.  The  primary  objective  of  this  thesis  research  was  to  determine  if  an 
enhanced  cannon  plug  presentation  method,  by  means  of  a  head  mounted  display,  would 
reduce  maintenance  technician’s  cannon  plug  pin  selection  time  and  errors. 

Research  Hypotheses 

The  overall  research  hypothesis  is  that  graphically  enhanced  data  presentations 
will  improve  technician  cannon  plug  pin  selection  and  connection  task  performance.  The 
following  hypotheses  detail  how  technician  performance  is  defined: 

1 .  Fewer  cannon  plug  pin  connection  errors  are  made  using  the  graphically 
enhanced  presentation  on  the  HMD  than  the  unenhanced  presentation. 

2.  Pin  selection/connection  takes  less  time  using  the  graphically  enhanced 
presentation  method  than  the  unenhanced  presentation. 

3.  Cannon  plugs  with  fewer  pins  will  have  fewer  connection  errors  than  cannon 
plugs  with  many  pins. 
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4.  Cannon  plugs  with  fewer  pins  will  take  less  time  to  complete  the  task  than 
cannon  plugs  with  many  pins. 


5.  For  enhanced  presentations,  there  is  no  statistically  significant  difference  in 
time  to  complete  the  tasks  between  cannon  plugs  with  many  and  few  pins. 

6.  For  unenhanced  presentations,  cannon  plugs  with  few  pins  will  take  less  time 
to  complete  the  tasks  than  carmon  plugs  with  many  pins. 

7.  For  enhanced  presentations,  there  is  no  statistically  significant  difference  in 
the  number  of  task  completion  errors  between  cannon  plugs  with  many  and 
few  pins. 

8.  For  unenhanced  presentations,  cannon  plugs  with  few  pins  will  have  less 
errors  in  task  completion  than  cannon  plugs  with  many  pins. 

Test  Equipment  and  Scope 

Armstrong  Laboratory  provided  the  hardware  and  software  for  use  in  this  research 
effort.  They  have  conducted  research  with  the  HMD  system  and  software  since  1992  and 
have  the  system  fully  integrated.  The  HMD  eyepiece  had  a  green  monochrome  640  x  480 
pixel  resolution  VGA  screen  and  was  made  by  Kopin™.  The  graphical  data  presentation 
was  programmed  in  Visual  Basic™  software  language. 

The  five  cannon  plug  connector  breakout  boxes  used  in  this  research  were  also 
supplied  by  Armstrong  Laboratory.  Figure  3  is  a  picture  of  the  four  break-out  boxes 
containing  the  four  different  camion  plug  types,  which  have  12  to  79  pins,  that  are  used  as 
the  test  instruments.  The  pins  are  visually  referenced  by  either  numbers  or  letters  printed 
on  the  plug  mating  surface.  The  wide  variance  in  pin  numbers  was  selected  to  test  the 
effect  of  the  increased  number  of  cannon  plug  pins  on  the  maintenance  technician  error 
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rate.  The  fifth  cannon  plug  connector  breakout  box,  shown  in  Figure  4,  is  rectangular  and 
is  used  for  training  and  HMD  orientation. 


"IliiififmtL 


Figure  3:  Picture  of  Breakout  Boxes  Used  in  this  Research 


Figure  4;  Picture  of  Practice  Breakout  Box  Used  in  this  Research 
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The  technical  manual  schematics  of  the  cannon  plugs  were  programmed  into 
Visual  Basic™  software  for  the  baseline  test  and  then  augmented  through  visual  queues 
and  contrast  schemes  for  the  enhanced  test.  Each  maintenance  technician  was  required  to 
complete  sixteen  total  tasks,  four  on  each  cannon  plug.  Half  of  the  tasks  were  completed 
using  the  unenhanced  graphical  presentation  and  half  used  the  enhanced  presentation 
method.  This  test  method  scheme  was  developed  using  a  randomized  block  design  that  is 
discussed  in  Chapter  Three  of  this  study. 

General  Approach 

A  literature  review  was  conducted  to  identify  the  relevant  aspects  of  this  research 
effort.  Maintenance  practices,  computerized  data  presentation  methods  (to  include  head 
mounted  display  devices),  monochrome  displays,  human  factors  in  visual  acuity  and 
fixation,  and  experimental  design  research  streams  were  investigated.  From  the  literature 
review,  the  experimental  method  and  data  analysis  technique  for  this  thesis  were 
developed.  Additionally,  the  applicability  to  previous  and  current  research  and  the 
significance  to  the  Air  Force  was  established. 

The  experimental  sampling  method  was  designed  around  the  test  equipment 
available  during  this  research  effort.  The  entire  system  was  assembled  and  the 
experimental  test  method  and  procedures  were  tested  and  validated  through  a  pre-test 
prior  to  gathering  the  test  data  in  the  field.  Data  was  then  gathered  fi’om  an  active  field 
unit.  The  data  was  then  analyzed  and  tested  against  the  thesis  research  objective  of  this 
study  —  to  see  if  the  enhanced  data  presentation  method  does,  in  fact,  reduce  the  cannon 
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plug  pin  identification  and  selection  time  and  error  rate.  The  results  of  this  analysis  are 
presented  in  Chapter  Five  of  this  study. 

Summary 

This  thesis  utilized  and  built  upon  research  previously  performed  by  Armstrong 
Laboratory  and  Master’s  candidates  from  the  Air  Force  Institute  of  Technology  (AFIT). 
Investigating  the  benefits  of  enhanced  graphics  and  head-mounted  displays  logically 
follows  field  observations  resulting  from  several  studies.  These  studies  demonstrated  the 
reduction  of  errors  using  computerized  maintenance  aids  verses  traditional  paper 
technical  references,  and  the  improved  performance  obtained  from  using  HMDs  over  flat 
screen  portable  computers. 

Thesis  Overview 

This  thesis  is  divided  into  five  chapters.  The  first  chapter  canvasses  the  general 
background  information  on  this  thesis’  research  focus,  method,  and  questions.  Chapter 
Two  comprises  a  literature  review  on  the  current  state  of  computer  enhanced  data 
presentation  methods  and  their  applicability  to  this  thesis.  Chapter  Three  outlines  the 
research  methodology  used.  Chapter  Four  is  the  analysis  of  the  data  that  was  obtained 
through  the  methods  described  in  Chapter  Three.  Chapter  Five,  discusses  the  results  of 
the  analysis  of  the  data  and  provides  recommendations  for  future  research. 
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//.  Literature  Review 


Overview 

This  chapter  comprises  a  literature  review  of  previous  head  mounted  display 
(HMD)  data  presentation  method  studies,  ways  to  optimize  the  presented  data,  and  HMD 
applicability  to  military  maintenance  operations.  The  literature  review  incorporates 
commercial,  professional,  and  trade  journals.  Department  of  Defense  technical  reports, 
the  World  Wide  Web,  and  previous  AFIT  theses. 

Armstrong  Laboratory  requested  that  the  monocular  head-mounted  display  from 
Kopin™  be  used  as  the  display  device  for  presenting  the  experimental  information  to  the 
test  subjects.  Because  of  this  request,  only  monocular  HMD  designs  were  investigated  in 
this  literature  review.  The  information  collected  was  limited,  but  did  include  information 
on  ways  to  improve  data  presentation  on  computer  screens  and  some  advantages  and 
disadvantages  in  the  use  of  an  HMD. 

Military  Maintenance  Applications 

Military  maintenance  technicians  face  many  unique  obstacles  to  performing  their 
assigned  duties  that  their  civilian  counterparts  do  not.  Deployments,  unique  settings, 
remote  locations,  and  a  wide  variety  of  equipment  are  but  a  few  examples  of  that 
difference.  Any  maintenance  aids  and  equipment  that  are  used  by  military  maintenance 
technicians  must  be  able  to  perform  in  many  imaginable  situations  and  must  be  highly 
reliable. 
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Automated  maintenance  aids  are  becoming  commonplace  in  the  military 
maintenance  community.  Maintenance  Error  Decision  Aids  (MED A),  Avionics 
Troubleshooting  Systems  (ATS),  Flight  Control  Maintenance  Diagnostic  Systems 
(FCMDS),  Aviation  Diagnostics  and  Maintenance  (ADAM)  systems,  and  Integrated 
Maintenance  Information  Systems  (IMIS)  are  but  a  few  of  the  computer-based  aircraft 
maintenance  aids  that  are  beginning  to  surface  on  Air  Force  flightlines  (Hibit  and  Marx, 
1994;  Gulick  and  Kell,  1993;  Schroeder,  Smith,  Bursch,  and  Meisner,  1992;  Le  Beau,  et. 
al.,  1991;  Thomas,  1995). 

These  systems  offer  the  power  and  speed  of  computer-driven  maintenance  tools 
with  the  knowledge  that  an  expert  system  brings.  Expert  systems  provide  any 
maintenance  technician  the  detailed  system  knowledge  usually  only  obtainable  through 
years  of  experience.  Expert  systems  can  enable  inexperienced  and  uncertain  technicians 
the  system  knowledge  to  operate  as  if  they  were  experienced.  These  systems  also  provide 
reminders  and  double-checks  to  more  experienced  technicians  when  they  encounter 
something  that  they  are  uncertain  about  in  the  system. 

Research  conducted  by  Friend  and  Grinstead  (1992)  indicates  that  HMDs  improve 
aircraft  maintenance  technician  inspection  task  performance  over  the  current  flat-screen 
presentations  that  are  available.  HMDs  can  be  incorporated  into  aircraft  maintenance 
training  to  improve  technician  performance  from  the  start  of  their  career.  The  days  of 
extensively  training  maintenance  technicians  and  investing  years  before  they  can  be 
allowed  to  perform  maintenance  tasks  can  be  in  the  past  (Basta,  1995).  Computer-based 
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training  using  HMDs  can  enable  first-time  technicians  to  perform  like  veterans  on  the 
flightline. 

Previous  HMD  Studies 

Through  extensive  literature  examination,  this  researcher  has  found  only  three 
evaluations  that  previously  performed  an  assessment  of  the  applicability  of  HMD 
technology  in  a  maintenance  environment.  The  first  evaluation  took  place  in  1990,  vvdth 
the  report  released  in  1991,  by  the  General  Dynamics  Electronics  Division.  General 
Dynamics  conducted  a  comparative  evaluation  of  HMD  and  flat-screen  computer 
technology.  F-16  aircraft  maintenance  technicians  were  asked  to  compare  HMDs  and 
hand-held  flat-screen  computers  while  performing  bench  checks  on  equipment.  The 
technicians  completed  a  rating  scale  and  remarked  on  open-ended  questions  (Edwards 
Evaluation  Report,  1991). 

The  General  Dynamics  study  resulted  in  the  following  conclusions: 

1 .  Both  HMD  and  flat-screens  are  suitable  for  displaying  technical  data  in 
a  static  environment.  However,  more  technicians  indicated  a 
preference  for  using  the  flat-screen  display  device  over  the  HMD. 

2.  Technician  display  presentation  method  preference  is  dependent  on  the 
task  they  are  performing.  Technicians  indicated  that  in  performing 
tasks  that  require  their  hands  to  be  free,  such  as  situations  where  tools 
are  in  both  hands  and  the  technical  data  must  be  in  close  proximity, 

HMDs  are  the  better  choice. 

A  flaw  in  the  study  may  have  been  that  the  test  participants  did  not  receive  enough  time 
to  adjust  to  wearing  and  using  a  bulky  and  heavy  (over  10  pounds)  HMD.  Advancements 
in  technology  enabled  the  use  of  an  HMD  that  weighed  less  than  one  pound  in  this  thesis. 
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Because  technicians  indicated  the  HMD  would  be  preferential  in  certain  situations, 
display  presentation  method  may  be  task  dependent. 

The  results  of  the  General  Dynamics  evaluation  helped  to  identify  the 
maintenance  tasks  to  be  performed  and  to  highlight  the  need  to  establish  the  HMD 
training  conducted  for  this  thesis.  The  present  research  has  test  subjects  perform  cannon 
plug  pin  selection  and  connection  tasks  that  require  their  hands  to  be  free.  Technicians 
have  to  hold  the  cannon  plugs  and  test  adapters  simultaneously,  they  have  no  free  hand  to 
hold  the  required  technical  data.  Technicians  also  conduct  familiarization  training  on  the 
HMD  and  complete  practice  tasks  before  the  collection  of  the  actual  test  data. 

Masquelier  (1991)  conducted  the  second  study  that  addressed  HMD  use  in  a  static 
maintenance  environment.  She  had  F-16  aircraft  maintenance  technicians  use  an  HMD 
and  a  flat-screen  computer  to  display  the  technical  information  for  routine  intermediate 
level  maintenance  bench-top  troubleshooting  tasks.  Masquelier  (1991)  found  no  overall 
statistically  significant  results  in  the  difference  in  performance  between  the  group  that 
used  the  HMD  and  the  group  that  used  the  flat-screen  display  device  to  display  the 
technical  information.  However,  she  did  find  that  the  aircraft  maintenance  technicians 
with  ‘experience’  statistically  performed  better,  took  less  time,  and  made  fewer  mistakes, 
than  those  technicians  with  limited  ‘experience.’  Masquelier  (1991)  defined  experience 
as  those  technicians  with  more  than  one  year  of  actual  hands-on  system  maintenance 
time. 
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The  findings  of  improved  performance  for  ‘experienced’  maintenance  technicians 

found  in  Masquelier’s  (1991)  study  supports  assertions  made  by  Heleander  (1988)  that: 

Users  who  have  acquired  extensive  knowledge  and  skill  related  to  their 
job  might  be  expected  to  use  a  computer  system  on  the  job  more 
effectively  than  users  with  little  domain  specific  knowledge. 

(Heleander,  1988) 

The  Masquelier  (1991)  and  Heleander  (1988)  studies  were  important  to  this  thesis  in  that 
they  supported  the  elimination  of  the  technician  experience  variable  from  the  study. 

This  thesis  was  limited  to  ‘experienced’  technicians  for  two  main  reasons.  One 
reason  was  to  eliminate  a  potentially  confoimding  variable.  The  other  was  that 
‘experienced’  technicians  are  familiar  with  the  task  being  performed  and  could  provide  a 
more  in-depth  analysis  of  the  HMD.  Inexperienced  technicians  would  be  trying  to  figure 
out  both  the  maintenance  task  and  the  HMD  during  the  experiment,  and  they  may  not 
have  been  able  to  provide  as  insightful  feedback  as  the  ‘experienced’  technicians. 

Friend  and  Grinstead  (1992)  conducted  a  third  study  that  used  an  HMD  in  a 
maintenance  environment.  They  looked  at  the  display  of  aircraft  maintenance  technical 
data  on  an  HMD  and  flat-screen  display  in  a  flightline  maintenance  environment.  An 
important  difference  between  Friend  and  Grinstead’s  (1992)  and  Masquelier’s  (1991) 
studies  is  that  technicians  in  Friend  and  Grinstead’s  (1992)  study  conducted  tasks  on  the 
flightline,  in  normal  working  conditions.  A-7D  aircraft  maintenance  technicians 
completed  an  operational  checkout  task  using  an  HMD  and  a  flat-screen  monitor  to 
display  the  maintenance  technical  data.  The  technicians  are  also  divided  into 
‘experienced’  and  ‘inexperienced’  groups. 
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Friend  and  Grinstead  (1992)  found  that  technicians  using  the  HMD  correctly 
completed  the  task  quicker  than  the  technicians  that  used  the  flat-screen  display.  In 
addition,  the  technicians  using  the  HMD  made  fewer  errors  than  the  technicians  using  the 
flat-screen  display.  Furthermore,  the  ‘experienced’  technicians  outperformed  the 
‘inexperienced’  technicians  in  both  task  completion  time  and  errors  committed  (Friend 
and  Grinstead,  1992).  The  results  of  Friend  and  Grinstead’s  (1992)  study  supported  the 
use  of  only  ‘experienced’  technicians  in  this  thesis. 

HMD  Disadvantages  and  Advantages 

Most  HMD  studies  have  been  conducted  in  dynamic  environments  —  in  aircraft 
cockpits  or  aircraft  simulators  —  and  have  identified  several  disadvantages  and 
advantages  in  their  use  over  traditional  flat-screen  displays.  The  primary  disadvantages 
are  the  wearer  side  effects  that  often  accompany  the  prolonged  use  of  HMDs.  HMD 
advantages  are  experienced  in  improved  user  mobility  and  increased  user  task 
effectiveness  as  measured  in  increased  performance  and  reduced  task  completion  time. 

HMD  Disadvantages.  The  side  effects  experienced  from  prolonged  HMD  use 
include  retinal  rivalry,  reduced  visual  resolution,  depth  perception  difficulty,  limited  field 
of  view,  and  increased  eye  stress  and  fatigue  (Rash  and  Martin,  1988;  Hale  and  Piccione, 
1990;  Haworth  and  Newman,  1993;  Kotulak  and  Morse,  1995).  Retinal  rivalry  is  the 
ability  to  selectively  switch  back  and  forth  between  the  two  images  being  presented  in 
separate  eyes  (Rash  and  Martin,  1988).  This  ability  is  important  for  the  use  of  a 
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monocular  HMD  in  which  the  wearer  must  do  exactly  that,  switch  between  the  HMD 
display  in  one  eye  and  the  task  being  performed  in  the  other. 

Visual  resolution,  depth  perception,  and  field  of  view  are  all  impacted  by  having 
the  HMD  directly  in  front  of  one  eye  and  nothing  in  front  of  the  other.  Newer  adjustable 
HMDs  can  be  physically  adjusted  to  allow  the  wearer  to  “look  through”  it  as  if  it  were  not 
there  (Haworth  and  Newman,  1993;  Kotulak  and  Morse,  1995).  Adjusting  the  HMD  may 
improve  some  effects,  but  it  often  leads  to  others.  Eye  strain  and  fatigue  often 
accompany  long-term  use  of  HMDs  and  are  even  more  prevalent  in  the  “look  through” 
designs  (Hale  and  Piccione,  1990).  These  factors  are  also  often  compounded  by  the 
effect  on  the  wearer’s  depth  perception  that  accompanies  HMD  use  (Hale  and  Piccione, 
1990). 

In  dynamic  environments,  HMD  wearers  must  not  only  adjust  to  having  a  display 
screen  directly  in  front  of  one  eye,  but  they  must  also  continue  to  monitor  their 
continually  changing  surroundings.  Attention  is  constantly  shifted  between  external 
stimuli  and  the  data  presented  on  the  HMD  screen.  As  the  wearer  becomes  tired  their 
eyes  are  less  able  to  adjust  as  quickly  and  the  problems  can  become  severe  (Kotulak  and 
Morse,  1995). 

The  General  Dynamics  (1991),  Masquelier  (1991),  and  Friend  and  Grinstead 
(1992)  studies  tested  the  HMD  in  more  static  environments.  These  studies  indicated  that 
users  did  experience  a  slight  degree  of  eye  strain  from  not  being  use  to  wearing  and  using 
the  HMD.  Participants  also  indicated  that  they  had  some  interference  between  the  HMD 
display  and  task  object  on  which  they  were  trying  to  focus.  The  test  participants 
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indicating  that  they  initially  had  these  problems,  also  indicated  that  after  they  became 
accustomed  to  wearing  the  HMD  they  were  able  to  eliminate  these  problems. 

None  of  the  studies  indicated  what  percentage  of  the  participants  experienced 
these  difficulties.  The  coverage  of  these  problems  in  the  reports  hinted  that  the  actual 
problems  were  minor  and  relatively  few  test  subjects  had  them.  This  research  effort  also 
asked  its  test  subjects  if  they  experienced  any  visual  problems  during  or  after  wearing  the 
HMD. 

HMD  Advantages.  The  basic  advantages  of  HMDs  are  that  they  are  light, 
compact,  inexpensive,  and  more  rugged  than  typical  flat-panel  displays.  When  attached 
to  a  portable,  wearable  computer  system,  the  HMD  enables  complete  user  mobility.  This 
mobility  allows  the  user  to  directly  interact  with  their  environment  while  using  the 
computer  and  HMD  as  a  task  performance  aiding  device  (Quill,  Kancler,  &  Masquelier, 
1995).  Mobility  provides  distinct  advantages  in  a  maintenance  environment  where 
technicians  must  be  able  to  access  tools,  test  equipment,  parts,  and  read  technical  data 
simultaneously. 

Additionally,  as  discussed  previously,  data  displayed  on  HMDs  improve 
technician  task  performance,  task  completion  time,  and  errors  made,  over  using  paper 
manuals  or  flat-screen  displays.  Improving  technician  performance  has  many  advantages. 
As  discussed  by  Ebling  (1997)  and  Langford  (1995),  reduced  maintenance  time  increases 
equipment  availability.  Technicians  are  also  able  to  be  more  efficient  and  accomplish 
more  during  a  normal  work  shift.  Accomplishing  tasks  quicker  and  making  technicians 


18 


more  efficient  may  enable  the  reduction  in  total  manpower  requirements  without  over 
stressing  an  already  limited  maintenance  manpower  pool. 

Equipment  costs  are  reduced  because  the  cost  of  labor  is  reduced  resulting  in  a 
total  life  cycle  cost  reduction  for  the  system.  Cost  reduction  is  the  key  to  a  system’s 
survival  in  this  time  of  budget  cuts  and  force  reductions.  If  the  cost  to  maintain  a  system 
can  be  reduced,  it  can  enable  the  military  to  absorb  possible  future  budget  cuts  without 
sacrificing  maintenance  effectiveness.  The  cost  savings  could  also  be  flirmeled  into  other 
needed  areas.  Either  way,  the  cost  savings  can  be  beneficial  to  the  entire  military. 

As  shown  in  the  Raaijmakers  and  Verduyn  (1996)  study,  technicians  with 
relatively  little  training  can  be  as  effective  as  experienced  technicians  when  presented 
with  maintenance  aids  that  enable  system  understanding  and  comprehensive  data 
presentation  for  task  performance.  HMDs  can  be  a  part  of  that  system.  These  types  of 
systems  can  reduce  the  skill  level  required  for  the  maintenance  technician  working  on  a 
system.  Reduced  skill  relates  to  reduced  training  which  equates  to  less  cost  to  prepare 
maintenance  technicians. 

Technical  Data  Presentation 

Computer  Screen  Design.  The  studies  cited  in  the  previous  sub-section,  all  used 
‘simple’  technical  data  presentation  methods.  Much  of  the  technical  data  presented  to  the 
maintenance  technicians  are  simply  digitized  representations  of  the  paper  products 
currently  being  used  or  simple  textual  troubleshooting  flow  charts.  The  paper  data  is 
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scanned  and  imported  into  computerized  versions  of  the  original  technical  data,  which 
enables  large  amounts  of  data  to  be  readily  available  to  the  maintenance  technicians. 

Previous  research  established  that  the  computerized  version  of  the  technical  data 
can  improve  aircraft  maintenance  technician’s  performance  by  reducing  the  technician’s 
total  task  completion  time  and  reducing  the  total  number  of  errors  made  (Becker,  1990; 
Nugent,  1987;  Thomas,  1995).  The  added  findings  from  General  Dynamics  (1991), 
Masquelier  (1991),  and  Friend  and  Grinstead  (1992)  indicate  that  using  an  HMD  is  more 
advantageous  over  a  flat-screen  display;  however,  errors  are  still  made. 

The  primary  objective  of  this  thesis  research  is  to  determine  if  maintenance 
technicians’  cannon  plug  pin  selection  and  connection  performance  could  be  improved 
through  the  use  of  an  enhanced  technical  data  presentation  method  on  an  HMD. 

Display  Characteristics.  Display  presentation  properties  for  graphical  and 
scientific  data  have  been  extensively  studied.  Factors  such  as  display  background,  item 
contrast,  viewing  distance,  graphic  size,  graphic  orientation,  and  amount  of  irrelevant 
information  on  the  display  have  long  been  known  to  be  areas  of  concern  for  graphically 
displayed  data.  When  a  technician  is  required  to  select  and  make  decisions  based  on 
graphically  displayed  data,  these  items  must  be  carefully  monitored  and  controlled. 

Pertinent  information  must  be  clearly  distinguishable  from  its  immediate 
background  (Krendel  and  Wodinsky,  1960).  Monochrome  displays  can  be  as  reliable  as 
color  displays  if  the  contrast,  spacing,  size,  and  density  are  controlled  (Monk  and  Brown, 
1975;  Smith,  1963;  Wagner,  1977).  In  fact,  monochrome  monitors  are  preferred  for 
facilitating  the  display  of  proper  images.  Monochrome  displays  allow  the  best  contrast 
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between  the  data  being  presented  and  display  background  (Heleander,  1988).  Display 
clutter  is  also  a  major  hindrance  to  correctly  locating  and  identifying  data.  If  information 
is  not  essential  to  the  situation,  it  must  be  eliminated  (Gordon,  1968;  Heleander,  1988; 
Yonas  and  Pittenger,  1973). 

This  thesis  utilized  the  Kopin™  HMD  which  has  a  monochrome  display.  The 
data  presented  in  the  study  was  limited  to  only  the  exact  information  needed  to  perform 
the  tasks  requested  in  the  test  program.  The  Kopin™  HMD  also  had  an  adjustable 
eyepiece  that  allowed  the  wearer  to  adjust  it  for  optimal  viewing  distance,  thereby 
eliminating  the  visual  acuity  problems  associated  with  viewing  displays  at  improper 
distances  (Giese,  1946;  Quill,  Kancler,  &  Batchelor,  1996). 

From  ensuring  the  basics  of  proper  data  display,  investigation  focus  switched  to 
ways  to  improve  the  displayed  data.  This  thesis  was  concerned  with  enhancing  the  data 
that  was  presented  to  improve  maintenance  technician  performance.  The  test  program 
utilized  in  this  research  presented  data  in  the  form  of  text  and  graphics. 

Previous  studies  have  looked  at  textual  information  and  how  it  is  presented. 
Textual  data  should  be  in  both  upper  and  lower  case  letters  for  quicker  and  more  accurate 
identification  (Heleander,  1988).  Studies  by  Bennett  &  Flach  (1992),  Boles  &  Wickens 
(1987),  and  Payne  &  Lang  (1995)  have  shown  that  mixed-format  displays,  graphics  and 
text  combined,  produce  better  performance  results  than  either  of  the  two  alone.  This 
finding  is  significant  as  most  military  technical  data  routinely  provides  either  text  or 
graphics.  The  graphics  that  usually  accompany  the  textual  data  are  unreadable.  It  is 
either  blurry,  shrunk  too  small  to  read,  or  vague  as  to  what  is  being  shown. 
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Wickens,  Merwin,  and  Lin  (1994)  conducted  a  study  on  graphical  representation 
of  data.  Their  study  indicated  that  2-D  graphics  with  ‘visual  cues’  provided  the  best 
performance  results  from  their  test  subjects.  The  visual  cues  provided  were  different 
colors,  bold  letters,  highlights,  and  shading.  Naval  engineers  were  tested  in  a  study  by 
Raaijmakers  and  Verduyn  (1996)  which  gave  them  unfamiliar  fault  problems  and  a 
simple  help  system  that  included  graphics  with  visual  cues.  Engineer  performance  in 
detecting  and  identifying  the  faults  increased  when  they  used  the  help  system. 

A  computer  screen  format  handbook  by  Galitz  (1985)  provides  an  additional  tip  in 

improving  the  readability  of  data  displayed  on  computer  screens.  The  handbook  states: 

Specific  areas  of  the  screen  should  be  reserved  for  certain  kinds  of 
information,  such  as  commands,  error  messages,  and  input  fields,  and 
maintain  these  areas  consistently  on  all  screens.  (Galitz,  1985) 

The  handbook  also  recommends  that  if  a  program  continually  displays  screens  with 

similar  information,  not  mentioned  above,  then  the  information  should  also  be  presented 

in  the  same  location  each  time  it  is  displayed.  Consistently  displaying  information  in  the 

same  location  enables  the  viewer  to  identify  data  more  quickly. 

This  thesis  incorporated  the  recommendations  and  findings  of  the  afore  mentioned 
references.  A  monochrome  HMD  with  an  adjustable  eyepiece  was  used  to  display  textual 
and  graphical  information.  The  presented  text  was  in  both  upper  and  lower  ease  letters. 
Only  the  information  required  to  perform  the  assigned  task  was  presented.  Additionally, 
all  screens  that  presented  the  same  or  similar  information  did  so  in  the  same  location  on 
the  display. 
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Summary 


Head  mounted  display  studies  have  shown  improved  maintenance  technician 
performance  over  traditional  flat-screen  display  methods.  With  careful  consideration  as 
to  what  data  is  displayed  and  how  the  data  is  displayed,  displays  can  be  arranged  to 
enhance  viewer  performance.  HMDs  do  inherently  contain  some  disadvantages  when 
used  in  a  dynamic  environment,  such  as  in  flight  in  an  aircraft  cockpit.  In  more  static 
environments,  such  as  in  maintenance  settings,  the  disadvantages  of  HMDs  can  be 
controlled  and  the  advantages  gained  provide  better  technician  performance.  Chapter 
Three  outlines  the  research  methodology  used  in  this  study. 
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III.  Methodology 


Research  Design  Background 

This  research  investigated  the  relationship  between  avionics  maintenance 
technician  performance  and  the  methods  used  to  display  the  technical  information.  Hie 
information  used  was  presented  on  a  single  display  type,  a  monocular  head-mounted 
display  (HMD),  but  varied  in  its  containment  of  enhancements  to  the  data  presentation 
(enhancements  verses  no  enhancements).  Research  conducted  by  Nugent  (1987) 
indicates  that  the  electronic,  computer-based,  presentation  of  technical  data  improves 
maintenance  technician  performance  over  the  traditional  paper  (technical  manuals)  data 
presentation  methods.  Friend  and  Grinstead  (1992)  conducted  research  that  indicates 
information  presented  on  a  head-mounted  display  improves  technician  performance  over 
computer-based,  flat  screen  laptop  computers.  This  research  investigated  whether  data 
presented  on  an  HMD  can  be  enhanced  to  improve  technician  performance  even  more. 
More  specifically,  the  present  study  investigated  the  effects  of  graphical  enhancements  of 
relatively  simple  and  complex  equipment  setups  on  maintenance  technician  performance. 

Experiment  Test  Subjects 

Avionics  maintenance  technicians  from  the  2d  Bomb  Wing  at  Barksdale  AFB,  LA 
were  the  test  subjects  in  this  experiment.  The  technicians  were  volunteers  that  were 
randomly  chosen  from  the  pool  of  available  personnel  at  the  base.  Criteria  for 
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consideration  in  the  available  pool  were  based  on  time  in  service,  time  on  the 
flightline/bench,  current  duties,  and  eyesight.  Technicians  had  to  have  a  minimum  of 
four  years  continuous  active  duty  experience,  must  have  been  working  on  the 
flightline/bench  for  at  least  two  years,  and  must  currently  be  performing  maintenance 
duties  on  the  flightline/bench.  Technicians  also  had  to  possess  at  least  20/20  corrected 
vision  and  could  not  wear  bifocals  or  trifocals  in  order  to  be  considered  in  the  available 
pool  of  technicians  for  this  study. 

The  technician  experience  requirements  were  designed  to  eliminate  the  experience 
variable  from  entering  into  the  analysis.  Masquelier  (1991)  and  Friend  and  Grinstead 
(1992)  both  showed  that  technician  experience  levels  have  a  signifieant  effect  on 
technician  performance.  The  test  subjects  were  screened  for  a  minimum  eommon 
experience  level  to  eliminate  varying  experience  levels  as  an  experiment  variable.  This 
minimum  common  experience  level  included  time  in  serviee,  time  on  station,  eontinuous 
time  on  the  flightline  or  test  bench,  and  current  duties. 

Technician  eyesight  was  also  screened.  The  HMD  does  present  problems  for 
people  who  wear  bifocals  or  trifocals,  so  they  were  eliminated  from  the  available  pool  of 
technicians.  Eyesight  corrected  to  20/20  was  also  used  as  a  minimum  baseline. 
Technician  vision  was  not  tested  for,  but  technicians  were  asked  if  they  had  eorreeted 
20/20  vision.  Perfect  eyesight  is  not  required  for  HMD  use,  but  eliminating  people  with 
eyesight  problems  also  eliminated  one  potentially  confounding  variable. 

Ocular  dominance,  or  the  dominant  sighting  eye,  of  the  technician  was  not  tested, 
nor  is  it  required  that  the  HMD  be  used  on  the  dominant  eye.  Experiments  indicate  that 
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while  performing  static  maintenance  tasks,  those  tasks  that  don’t  require  technicians  to 
move  around  —  like  bench  testing,  technician  performance  is  not  affected  by  which  eye 
views  the  HMD  (Quill  et.  ah,  1996).  The  study  results  do  suggest  that  switching  the 
HMD  from  eye  to  eye  during  task  performance  does  affect  overall  technician  task  time 
performance.  Because  of  this,  technicians  performing  this  experiment  were  discouraged 
from  switching  the  HMD  from  eye  to  eye  in  the  middle  of  a  task. 

Experiment  Hardware  and  Software 

There  were  several  pieces  of  equipment  used  in  this  research.  The  HMD  was  a 
commercially  available  monocular  display  attached  to  a  headband  and  is  made  by 
Kopin™.  A  486,  Pentium™  controlled  portable  laptop  computer  was  used  by  the  test 
administrator  as  the  CPU,  memory,  and  control  device.  The  test  apparatus  were  five 
multi-pin  cannon  plug/breakout  box  assemblies.  The  four  assemblies  used  for  the 
experiment  were  circular.  The  fifth  assembly,  which  was  used  for  practice,  orientation, 
and  training,  was  rectangular. 

The  software  used  in  the  experiment  was  a  locally  developed  test  program 
developed  using  Visual  Basic™  4.0  language.  The  technicians  were  presented  a  menu- 
driven  test  program  that  provided  visual  cues  to  follow.  The  test  administrator  stepped 
through  the  program  using  a  laptop  computer  as  the  technician  indicated  he  or  she  was 
ready  to  continue. 
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Experiment  Design 


Identifying  the  proper  experiment  design  involved  two  basic  steps.  The  first  was 
defining  what  was  to  be  measured  and  the  second  was  identifying  the  factors  — 
identifying  characteristics  that  differentiate  the  treatments  from  one  another  used  in 
designing  the  experiment  (Montgomery,  1976).  From  these,  the  research  methodology 
was  developed.  This  study  focused  on  two  measures:  task  completion  time  and  error 
rate.  Additionally,  each  technician’s  opinions  and  perceptions  were  collected  through  a 
post-test  questionnaire  and  interview  to  supplement  the  quantitative  analysis.  Sample 
questions  are  included  in  Appendices  A  and  B.  The  factors  for  this  research  were  the 
number  of  pins  on  the  cannon  plug  and  data  display  presentation  method. 

There  were  four  different  circular  cannon  plugs  used  in  this  research,  each  had  a 
different  number  of  “female”  or  recessed  pins:  12, 13,  55,  and  79.  Female  cannon  plugs 
were  exclusively  chosen  to  reduce  task  variance,  and  because  the  indexing  and  visual 
references  are  more  consistently  marked  and  readable.  The  cannon  plugs  were  divided 
into  two  groups:  many  pins  and  few  pins.  The  cannon  plugs  with  12  and  13  pins  were  in 
the  ‘few’  pins  group  and  the  cannon  plugs  with  55  and  79  pins  were  in  the  ‘many’  pins 
group.  The  wide  variance  in  pin  numbers  was  selected  to  test  the  effect  of  the  increased 
number  of  carmon  plug  pins  on  the  maintenance  technician  error  rate.  Figure  5  shows 
drawings  of  the  front  view  of  the  few  pin  cannon  plug  layouts  and  Figure  6  shows 
drawings  of  the  front  view  of  the  many  pin  cannon  plug  layouts. 
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Figure  5:  Graphics  of  Cannon  Plug  Faces  with 
‘Few’  Pins  Used  in  this  Research 


Figure  6:  Graphics  of  Cannon  Plug  Faces  with 
‘Many’  Pins  Used  in  this  Research 


The  cannon  plugs  were  visually  referenced  either  alphabetically  or  numerically. 
There  was  one  numerically  indexed  and  one  alphabetically  indexed  cannon  plug  in  each 
of  the  groups.  The  indexing  scheme  of  the  cannon  plugs  was  not  being  tested.  The 
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different  indexing  types  were  seleeted  to  be  a  representative  sample  of  what  teehnicians 
encounter  in  the  field. 

This  experiment  used  four  different  pin  combinations  for  each  caimon  plug.  A 
random  number  generator  was  used  to  select  the  first  pin  combination  for  each  cannon 
plug.  The  remaining  three  combinations  were  selected  based  on  inter-pin  distance  and 
relative  location  on  the  cannon  plug.  This  pin  selection  scheme  was  done  to  keep  the  pin 
selection  and  connection  tasks  equivalent  for  each  of  the  four  trials  for  each  cannon  plug. 
Keeping  the  tasks  equivalent  eliminated  task  difference  as  a  possible  source  of  control 
error  in  the  experiment.  Table  1  shows  what  connector  type  and  pins  were  used  for  each 
of  the  sixteen  test  conditions. 


Table  1:  Task  Condition  Identification 


1  Task  Conditions 

Connector  Type 
(#  of  Pins) 

Pins  to  Connect  | 

Enhanced 

Unenhanced 

First 

Second 

E9a 

79 

24 

46 

E5a 

55 

N 

s 

E3a 

13 

9 

10 

E2a 

12 

A 

F 

E9b 

79 

15 

59 

E5b 

55 

F 

X 

E3b 

13 

4 

12 

E2b 

U2b 

12 

D 

J 

E9c 

U9c 

79 

6 

54 

E5c 

U5c 

55 

C 

V 

E3c 

U3c 

13 

2 

11 

E2c 

U2c 

12 

E 

K 

E9d 

U9d 

79 

47 

67 

E5d 

U5d 

55 

S 

h 

E3d 

U3d 

13 

5 

6 

E2d 

U2d 

12 

D 

H 

Note:  These  cannon  plugs  are  circular.  The  cannon  plugs  differ  by 
the  number  of  connector  pins  they  have  and  their  pin  layout. 
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The  technical  data  presentation  method  was  either  enhanced  or  unenhanced.  Both 
presentations  contained  the  same  basic  information  that  is  presented  in  typical  technical 
manuals:  a  basic  picture  of  the  cannon  plug  in  question,  index  references,  and  a  table  of 
which  cannon  plug  pins  to  connect.  The  enhanced  presentation  contained  additional 
visual  cues  that  highlighted  or  emphasized  which  pins  to  coimect,  which  were  not 
provided  in  the  unenhanced  presentation.  Figure  7  contains  examples  of  both 
presentation  methods  for  the  55  pin  cannon  plug.  Appendix  C  contains  drawings  of  the 
HMD  presented  data  for  all  test  conditions. 


Unenhanced  HMD  Presentation 


Enhanced  HMD  Presentation 


55  Contacts 


From 

To 

55  Contacts 


Connect 

From 

D 

55  CONTACTS 


55  CONTACTS 


Figure  7:  Enhanced  and  Unenhanced  HMD  Presentation  Methods 


The  research  methodology  used  was  a  within-subject  mixed  two-variable/factor 
randomized  block  design  (Keppel  &  Zedeck,  1989).  The  within-subject  factor  was  the 
HMD  presentation  method,  enhanced  verses  unenhanced.  All  technicians  received  both 
the  enhanced  and  unenhanced  presentations  of  the  four  cannon  plugs,  twice.  This  design 
resulted  in  each  technician  performing  1 6  trials  in  each  treatment  or  task.  Sixteen  tasks 
per  technician  were  chosen  to  improve  the  power  of  the  analysis  of  the  experiment 
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results.  A  minimum  of  eight  tasks  were  required  so  that  each  technician  would  receive  all 
of  the  experimental  conditions.  Task  numbers  were  increased  by  eight  in  order  to  present 
all  the  experimental  conditions  for  data  comparison  analysis.  Learning  effect  was 
minimized  when  doubling  the  tasks  because  of  the  randomized  block  presentation  order 
(Keppel  &  Zedeck,  1989). 

The  presentation  order  was  obtained  through  a  randomized  block  manner.  Block 
randomization  was  done  to  avoid  data  presentation  order  effects.  It  was  selected  over 
other  data  display  methods  as  it  still  provided  presentation  order  control,  without 
requiring  counterbalancing,  and  allowed  more  generalizable  results  (Keppel  &  Zedeck, 
1989).  This  placed  no  limits  on  the  minimum  number  of  test  subjects  required  for  a 
significant  statistical  analysis.  The  block  randomization  was  accomplished  as  follows: 

(1)  The  16  tasks  to  be  performed  by  each  technician  were  divided  into  groups  of 
eight,  containing  two  of  each  cannon  plug  type  with  one  of  each  of  the 
presentation  methods,  enhanced  and  unenhanced. 

(2)  Each  group  of  eight  tasks  was  then  divided  in  half  Each  half  contained  one  of 
each  of  the  four  cannon  plug  types  of  either  presentation  method. 

(3)  Tasks  were  then  randomly  assigned  according  to  rules  one  and  two  above. 

Table  13  in  Appendix  D  shows  the  task  order  for  the  28  test  subjects.  A  Grecko- 

Latin  square  design  was  initially  utilized;  but  a  pilot  study  revealed  that  it  appeared  to 
have  an  order  affect  present,  so  it  was  replaced  with  the  block  randomized  design.  A 
second  pilot  study  run  using  the  block  randomized  design  showed  no  hint  of  an  order 
affect  present. 
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Experiment  Procedure 


Technicians  initially  completed  a  personal  background  form  to  ensure  that  they 
met  the  requirements  to  be  included  in  the  test  sample.  An  example  of  the  personal 
background  form  is  provided  in  Appendix  E.  The  technician  then  completed  a  Human 
Use  Research  Committee  (HURC)  form.  This  form  explains  the  technician’s  rights  and 
the  test  administrator’s  responsibilities  to  the  technician  during  the  test.  An  example  of 
the  form  is  provided  in  Appendix  F. 

Technicians  then  received  an  initial  briefing  on  what  they  would  do  during  the 
experiment  and  initial  training  on  the  use  of  the  HMD.  The  technicians  completed  a 
practice  session  using  a  rectangular  cannon  plug.  During  the  practice  session,  technicians 
were  exposed  to  the  exact  procedures  used  during  the  experiment. 

Before  starting  any  of  the  tasks,  the  technicians  were  allowed  to  orient  themselves 
on  the  task  to  be  performed  (cannon  plug  pin  layout  and  which  test  adapters  to  use).  This 
was  done  to  provide  a  common  starting  point  for  the  tasks.  The  technicians  were 
reminded  that  this  evaluation  was  concerned  with  their  task  performance  and  not  the 
actual  operation  of  the  HMD.  If  a  technician  was  experiencing  difficulty  in  operating  the 
HMD  or  any  of  its  components,  the  experimenter  helped  the  technician  in  the  proper 
equipment  operation  prior  to  beginning  a  task.  If  the  HMD  or  any  of  its  related 
components  failed  during  the  test,  or  the  technician  had  to  stop  the  test  for  any  reason, 
that  test  was  aborted  and  the  technician’s  data  was  eliminated  from  the  analysis. 
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For  the  experiment,  each  technician  performed  sixteen  measured  circular  cannon 
plug  pin  selection/connection  tasks  wearing  the  HMD.  Eight  of  the  tasks  were  completed 
using  the  graphically  enhanced  presentation  and  eight  were  completed  using  the 
unenhanced  presentation.  Technicians  were  allowed  only  one  selection/connection 
attempt  per  task.  The  error  rate  was  calculated  based  on  the  technician’s  ability  to 
correctly  identify  the  proper  pins  the  first  time.  To  control  for  data  collection  variance, 
the  same  experimenter  conducted  all  the  data  collection  activities. 

The  tasks  were  simple  by  design:  the  technicians  inserted  test  adapter  connectors 
in  two  specific  cannon  plug  pins.  The  presentation  on  the  HMD  specified  which 
connector  to  use  and  which  pins  to  coimect.  The  test  administrator  observed  the 
technician  as  the  tasks  were  performed  and  recorded  any  observations  on  an  experimenter 
observation  form.  An  example  of  the  form  is  provided  in  Appendix  F.  After  the 
technician  completed  each  task,  the  test  administrator  checked  to  see  if  it  was  completed 
correctly  and  also  recorded  the  results  on  the  experimenter  observation  form. 

The  task  time  was  automatically  recorded  in  the  test  program  as  the  task  was 
started  and  stopped  by  the  experimenter  at  the  direction  of  the  technician.  The  test 
administrator  controlled  the  test  program  advancement  through  the  use  of  a  portable 
laptop  computer.  The  HMD  worn  by  the  technician  was  tethered  to  the  laptop  computer. 
It  presented  the  program  data  and  visual  cues  for  the  technician  to  tell  the  test 
administrator  to  continue  the  program,  this  allowed  the  technician  to  concentrate  on  the 
cannon  plug  pin  selection  and  connection  tasks.  The  experiment  was  designed  to  test 
technician  cannon  plug  pin  selection  performance  using  an  HMD,  not  how  to  operate  a 
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computer.  Working  as  quickly  and  as  accurately  as  possible  was  emphasized  to  all  the 
technicians,  but  there  was  no  time  limit  placed  on  the  task  completion  time.  The  skill 
level  required  to  accomplish  the  tasks  was  such  that  time  limits  were  not  needed  and 
would  unnecessarily  complicate  the  analysis  of  the  test  data. 

Experimenter  control  was  achieved  through  standard  procedures  for  briefing, 
training,  interviewing,  and  debriefing.  Examples  of  the  scripted  presentations  are  in 
Appendices  B,  H,  I,  and  J.  All  test  subjects  received  the  same  instructions  and  guidance. 

The  collected  data  was  analyzed  using  a  repeated  measure  analysis  of  variance 
(ANOVA)  procedure  on  SPSS™  7.5  for  Windows™.  The  data  was  investigated  to  see  if 
there  was  a  statistically  significant  difference  between  the  unenhanced  and  enhanced 
graphical  display  methods  on  technician  task  completion  time  and  error  rate.  Appendix  K 
contains  the  raw  test  data  that  was  collected  for  this  study.  Chapter  Four  contains  the 
data  results  and  analysis  of  the  collected  test  data. 

Experiment  Hypotheses 

The  overall  research  hypothesis  was  that  graphically  enhanced  data  presentations 
would  improve  technician  cannon  plug  pin  selection  and  connection  task  performance. 

The  following  hypotheses  detail  how  technician  performance  was  defined: 

1 .  Fewer  cannon  plug  pin  connection  errors  are  made  using  the  graphically 
enhanced  presentation  on  the  HMD  than  the  unenhanced  presentation. 

2.  Pin  selection/connection  takes  less  time  using  the  graphically  enhanced 
presentation  method  than  the  unenhanced  presentation. 
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3.  Cannon  plugs  with  fewer  pins  will  have  fewer  connection  errors  than  cannon 
plugs  with  many  pins. 

4.  Cannon  plugs  with  fewer  pins  will  take  less  time  to  complete  the  task  than 
cannon  plugs  with  many  pins. 

5.  For  enhanced  presentations,  there  is  no  statistically  significant  difference  in 
time  to  complete  the  tasks  between  cannon  plugs  with  many  and  few  pins. 

6.  For  unenhanced  presentations,  cannon  plugs  with  few  pins  \vill  take  less  time 
to  complete  the  tasks  than  cannon  plugs  with  many  pins. 

7.  For  enhanced  presentations,  there  is  no  statistically  significant  difference  in 
the  number  of  task  completion  errors  between  cannon  plugs  with  many  and 
few  pins. 

8.  For  unenhanced  presentations,  cannon  plugs  with  few  pins  will  have  less 
errors  in  task  completion  than  cannon  plugs  with  many  pins. 


Experiment  Controls 


The  experimental  plan,  contained  in  Appendix  C,  was  developed  for  the 
experimenter  to  follow  during  the  data  collection.  The  plan  was  used  to  standardize  the 
briefing,  training,  and  task  performance  instructions  presented  for  all  the  technicians. 

This  kept  the  variations  from  these  areas  to  a  minimum.  This  evaluation  was  concerned 
with  cannon  plug  pin  selection/connection  error  rates  and  task  completion  times  and  not 
the  actual  operation  of  the  HMD.  Additionally,  the  following  previously  mentioned 
actions  were  performed  to  control  for  experiment  variation: 

1 .  Personnel.  Test  subjects  were  randomly  selected  from  the  available  pool  of 
avionics  technicians  at  the  2d  Bomb  Wing  at  Barksdale  AFB,  LA.  The 
technicians  had  a  minimum  of  four  years  active  duty  Air  Force  time  in  service, 
two  years  flightline/bench  experience,  and  their  current  jobs  were  on  the 
flightline/bench.  The  technicians  also  had  at  least  20/20  corrected  vision  and 
did  not  wear  bifocals  or  trifocals. 
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2.  Equipment.  If  the  HMD  or  any  of  its  related  components  failed  during  the 
test,  or  the  technician  had  to  stop  the  test  for  any  reason  that  test  was  aborted 
and  the  technician’s  data  was  eliminated  from  the  analysis.  If  a  technician 
experienced  difficulty  in  operating  the  HMD  or  any  of  its  components  the 
experimenter  helped  the  technician  in  the  proper  equipment  operation  prior  to 
beginning  the  task. 

3.  Procedures.  The  same  experimenter  conducted  all  the  data  collection 
activities.  The  task  performance  time  was  computed  by  the  computer  program 
itself  The  test  subjects  were  allowed  to  orient  themselves  on  the  task  to  be 
performed  before  beginning  the  test  (cannon  plug  pin  layout  and  which  test 
adapters  to  use).  Technicians  were  allowed  only  one  selection/connection 
attempt  per  task.  There  was  no  set  time  limit  for  task  completion. 

Summary 


This  chapter  outlined  the  research  methodology  and  experiment  controls  used  to 
conduct  this  experiment.  The  experimental  design,  to  include  the  tasks  performed, 
experiment  factors,  and  experiment  measures  used,  was  discussed.  The  experiment 
factors  were  the  number  of  pins  on  the  cannon  plug  (few  verses  many)  and  the  display 
presentation  method  (enhanced  verses  unenhanced).  The  experiment  measures  were  task 
completion  time  and  technician  error  rate  in  selecting  the  proper  cannon  plug  pins. 
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IV.  Results  and  Analysis 


Overview 

Twenty-eight  aircraft  avionics  maintenance  technicians  stationed  at  Barksdale 
AFB,  LA  each  performed  16  cannon  plug  pin  selection  and  coimection  tasks  for  this 
study.  Quantitative  and  qualitative  data  were  collected  from  all  of  the  participating  test 
subjects.  The  quantitative  data  are  analyzed  and  discussed  in  this  chapter.  The 
qualitative  data  that  were  collected  for  all  of  the  28  participating  technicians  through  the 
user  evaluation  questionnaire  and  the  structured  interview  pertain  only  the  usability  of  the 
HMD  system  and  are  discussed  in  Chapter  5. 

Four  of  the  technician’s  quantitative  data  sets  were  excluded  from  analysis.  One 
was  discarded  because  the  testing  procedure  was  compromised  while  the  technician  was 
performing  the  experimental  tasks.  The  technician  was  given  the  wrong  cannon  plug 
during  two  of  the  tasks  resulting  in  erroneous  task  completion  time  data  being  recorded. 
The  quantitative  data  was  determined  unusable  because  of  the  procedural  error. 

A  second  technician’s  data  was  eliminated  because  the  test  program  sequencing 
was  improperly  programmed  and  the  wrong  graphical  presentations  were  presented  to  the 
technician  during  the  test.  The  technician  performed  the  tasks  properly,  but  the  wrong 
presentation  resulted  in  the  experimental  design  being  compromised. 

The  other  two  data  sets  were  discarded  because  both  of  the  technicians  indicated 
during  the  structured  interview  that  they  did  not  see  the  enhancements  on  the  HMD 
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presented  graphics.  It  was  verified  that  they  did  not  see  the  enhancements  by  showing  the 
technicians  pictures  of  the  enhanced  graphical  presentations.  Consequently,  because  they 
did  not  see  the  enhanced  presentations,  they  did  not  use  them.  Thus,  the  data  collected 
for  the  enhanced  test  conditions  is  not  valid  and  could  not  be  used  for  analysis. 

The  quantitative  data  analysis  was  performed  on  the  remaining  24  sets  of  data, 
each  with  16  tasks.  Eight  of  the  tasks  were  completed  using  the  unenhanced  graphical 
presentation  method  and  the  other  eight  were  performed  using  the  enhanced  graphical 
presentation  method.  Task  completion  time  and  task  error  rate  were  collected  during  the 
study.  This  resulted  in  384  data  points  being  analyzed  for  both  task  completion  time  and 
task  error  rate  —  192  unenhanced  data  sets  and  192  enhanced  data  sets. 

The  quantitative  data  were  analyzed  using  a  repeated  measure  analysis  of  variance 
(ANOVA)  procedure.  The  following  sections  provide  the  collected  raw  test  data  for  the 
mean  task  completion  time  and  task  error  rates.  Additionally,  the  results  from  the  tests  of 
the  ANOVA  analysis  of  the  eight  proposed  research  hypotheses  and  the  interaction 
between  the  overall  average  task  completion  time  and  overall  error  rate  and  the 
unenhanced  and  enhanced  graphical  presentation  methods  follow  the  raw  data. 

Research  Data 

This  section  details  the  raw  quantitative  research  data  collected  during  the  course 
of  this  thesis  effort.  The  mean  task  completion  times  are  presented  first,  followed  by  the 
raw  error  data. 
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Task  Completion  Time.  Table  2  and  Figure  8  present  the  mean  task  completion 
time  data  for  the  24  qualified  test  subjects.  The  data  is  presented  for  the  ‘few’  pin  cannon 
plug  group,  ‘many’  pin  cannon  plug  group,  and  all  the  cannon  plugs  combined.  The  raw 
data  seems  to  show  a  difference  in  the  task  completion  times  between  the  unenhanced  and 
enhanced  graphical  presentation  methods  and  cannon  plug  pin  groups.  However,  the 
variance  of  the  data  groups  is  quite  large.  These  high  variances  prevent  any  conclusions 
being  drawn  from  the  raw  data  alone.  ANOVA  techniques  are  applied  later  in  this 
chapter  to  further  test  the  collected  data  against  the  eight  proposed  research  hypotheses. 


Table  2;  Mean  Task  Completion  Times 


Cannon  Plug 

#  of 

T  im  e  (s  e  c  )  | 

Connector  Group 

D  a  ta  P  o  in  ts 

Mean 

S  td  .  Dev. 

Few  Pins  (12  &  13): 

192 

1  2.23 

4.98 

Unenhanced: 

96 

13.25 

5.43 

Enhanced: 

96 

1  1  .21 

4.27 

Many  Pins  (55  &  79): 

192 

31  .31 

21.12 

Unenhanced: 

96 

23.38 

Enhanced: 

96 

BEBI 

1  4.59 

All  4  Cannon  Plugs: 

384 

21  .77 

1  8.06 

Unenhanced: 

192 

26.40 

21  .46 

Enhanced: 

1  92 

17.14 

1  2.26 

Figure  8;  Mean  Task  Completion  Times  Across  the  Various  Cannon  Plug  CategoriOes 
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Task  Error  Rate.  Table  3  and  Figure  9  present  the  task  error  rate  data  for  the  24 
qualified  test  subjects.  The  data  is  presented  for  the  ‘few’  pin  cannon  plug  group,  ‘many’ 
pin  cannon  plug  group,  and  all  the  cannon  plugs  combined.  The  raw  data  seems  to 
indicate  a  difference  in  the  task  error  rates  between  the  unenhanced  and  enhanced 
graphical  presentation  methods  and  cannon  plug  pin  groups.  ANOVA  techniques  are 
applied  later  in  this  chapter  to  further  test  against  the  eight  proposed  research  hypotheses. 


Table  3:  Task  Error  Rates 


Cannon  Plug 

#of 

#of 

Error  Rate 

Connector  Group 

Data  Points 

Errors 

(%) 

Few  Pins  (12  &  13): 

192 

17 

8.85 

Unenhanced: 

96 

12 

12.50 

Enhanced: 

96 

5 

5.21 

Many  Pins  (55  &  79): 

192 

52 

27.08 

Unenhanced: 

96 

43 

44.79 

Enhanced: 

96 

9 

9.38 

All  4  Cannon  Plugs: 

384 

17.97 

Unenhanced: 

192 

28.65 

Enhanced: 

192 

14 

7.29 

69 


Few-Pins  Many-Pins  All  Cannon  Plugs 


Cannon  Plug  Categories 


Figure  9:  Task  Error  Rates  Across  the  Various  Cannon  Plug  Categories 
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Research  Hypotheses 


Hypothesis  I.  This  hypothesis  predicted  that  fewer  cannon  plug  pin  connection 
errors  would  be  made  using  the  graphically  enhanced  presentation  over  the  unenhanced 
presentation.  Technicians  completed  a  total  of  384  tasks  —  1 92  tasks  using  the 
graphically  enhanced  presentation  and  192  using  the  unenhanced  presentation.  The  raw 
data  showed  that  technicians  committed  14  errors  when  they  used  the  enhanced 
presentation  method  to  perform  the  tasks  and  55  errors  when  they  used  the  unenhanced 
presentation  method.  This  result  appeared  to  provide  initial  support  to  the  hypothesis. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  4.  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  unenhanced  and  enhanced 
presentation  method  error  rates.  A  statistically  significant  result  was  observed.  The  F- 
calculated  value  of  20.26  exceeded  the  F-critical  value  of  4.28  and  thus  the  null 
hypothesis  was  rejected.  The  two  presentation  methods  did  not  have  the  same  error  rate. 

The  initial  data  and  the  ANOVA  analysis  support  the  hypothesis  that  the 
enhanced  presentation  method  would  have  a  lower  error  rate  than  the  unenhanced 
presentation  method. 


Table  4:  Overall  Task  Error  Data  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Value 

Calculated 

F-value 

P-Value 

Display  Type 
Error 

3.190 

3.622 

1 

23 

IH 

4.280 

20.255 

0.000 
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Hypothesis  II.  This  hypothesis  predicted  that  the  cannon  plug  pin  selection  and 
connection  task  time  would  be  shorter  using  the  graphically  enhanced  presentation  over 
the  unenhanced  presentation.  Technicians  completed  a  total  of  384  tasks  —  192  tasks 
using  the  graphically  enhanced  presentation  and  1 92  using  the  unenhanced  presentation. 
The  raw  data  showed  that  technicians  completed  the  tasks  quicker  using  the  enhanced 
presentation  (17.14  sec)  than  the  unenhanced  presentation  (26.40  sec).  However,  the 
standard  deviations  were  so  large  (12.26  sec  and  21 .46  sec  respectively)  that  no  initial 
conclusions  could  be  drawn  on  the  validity  of  the  hypothesis. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  5.  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  unenhanced  and  enhanced 
presentation  method  mean  task  completion  times.  A  statistically  significant  result  was 
obser\'ed.  The  F-calculated  value  of  92.44  exceeded  the  F-critical  value  of  4.28  and  thus 
the  null  hypothesis  was  rejected.  The  two  presentation  methods  did  not  have  the  same 
mean  task  completion  times. 

The  analysis  supports  the  hypothesis  that  the  enhanced  presentation  method 
would  take  less  time  to  perform  the  tasks  than  the  unenhanced  presentation  method. 


Table  5:  Overall  Mean  Task  Time  Data  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Value 

Calculated 

F-value 

P-Value 

Display  Type 
Error 

34941.586 

8693.727 

1 

23 

34941.586 

377.988 

4.280 

92.441 

0.000 
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Hypothesis  III.  This  hypothesis  predicted  that  tasks  performed  with  caimon  plugs 
with  ‘few’  pins  (the  12  and  13  pin  connectors)  would  have  fewer  errors  than  tasks 
performed  with  cannon  plugs  with  ‘many’  pins  (the  55  and  79  pin  connectors). 
Technicians  completed  a  total  of  384  tasks  —  192  tasks  utilizing  the  cannon  plugs  with 
few  pins  and  192  utilizing  the  cannon  plugs  with  many  pins.  The  raw  data  showed  that 
technicians  committed  17  errors  when  they  performed  the  tasks  on  the  ‘few’  pin  cannon 
plugs  and  52  errors  they  performed  the  tasks  on  the  ‘many’  pin  cannon  plugs.  This  result 
appeared  to  provide  initial  support  to  the  hypothesis. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  6.  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  ‘few’  pin  cannon  plug  and  the 
‘many’  pin  cannon  plug  task  completion  error  rates.  A  statistically  significant  result  was 
not  observed.  The  F-calculated  value  of  1.21  did  not  exceeded  the  F-critical  value  of 
4.28  and  thus  the  null  hypothesis  could  not  be  rejected. 

The  ANOVA  analysis  counters  the  hypothesis  that  technicians  would  commit 
fewer  errors  when  using  the  cannon  plugs  with  ‘few’  pins  than  the  cannon  plugs  with 
‘many’  pins.  However,  failing  to  reject  the  null  hypothesis  is  not  conclusive  —  further 
research  is  required  for  better  confirmation  of  this  analysis. 


Table  6:  ‘Few’  vs.  ‘Many’  Pin  Cannon  Plug  Overall  Task  Error  Data  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Value 

Calculated 

F-value 

P-Value 

Few  vs  Many 
Error 

1 

23 

1^ 

4.280 

1.205 

0.284 
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Hypothesis  IV.  This  hypothesis  predicted  that  the  task  time  of  cannon  plugs  with 
‘few’  pins  (the  12  and  13  pin  connectors)  would  be  shorter  than  the  task  time  of  cannon 
plugs  with  ‘many’  pins  (the  55  and  79  pin  connectors).  Technicians  completed  a  total  of 
384  tasks  —  192  tasks  utilizing  the  cannon  plugs  with  ‘few’  pins  and  192  utilizing  the 
cannon  plugs  with  ‘many’  pins.  The  raw  data  showed  that  technicians  completed  the 
tasks  more  quickly  on  the  ‘few’  pin  cannon  plugs  (12.23  sec)  than  on  the  ‘many’  pin 
cannon  plugs  (3 1 .3 1  sec).  However,  the  standard  deviations  were  too  large  (4.98  sec  and 
21 .12  sec  respectively)  to  allow  an  initial  conclusion  to  be  drawn  on  the  accuracy  of  the 
hypothesis. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  7.  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  ‘few’  pin  cannon  plug  and  ‘many’ 
pin  cannon  plug  mean  task  completion  times.  A  statistically  significant  result  was 
observed.  The  F-calculated  value  of  44.79  exceeded  the  F-critical  value  of  4.28  and  thus 
the  null  hypothesis  was  rejected.  The  two  cannon  plug  types  did  not  have  the  same  mean 
task  completion  times. 

The  analysis  supports  the  hypothesis  that  tasks  on  the  ‘few’  pin  cannon  plugs  take 
less  time  to  perform  than  the  tasks  on  the  ‘many’  pin  cannon  plugs. 


Table  7:  ‘Few’  vs.  ‘Many’  Pin  Cannon  Plug  Overall  Mean  Task  Time  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Value 

Calculated 

F-Value 

P-Value 

Few  vs  Many 
Error 

8223.253 

4223.060 

1 

23 

8223.253 

183.611 

4.280 

44.786 

0.000 
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Hypothesis  V.  This  hypothesis  predicted  that  for  enhanced  graphical 
presentations  only,  there  would  be  no  difference  between  the  task  times  of  cannon  plugs 
with  ‘few’  pins  (the  12  and  13  pin  connectors)  and  cannon  plugs  with  ‘many’  pins  (the  55 
and  79  pin  connectors).  Technicians  completed  a  total  of  192  enhanced  tasks  —  96 
utilizing  the  cannon  plugs  with  ‘few’  pins  and  96  utilizing  the  cannon  plugs  with  ‘many’ 
pins.  The  raw  data  showed  that  technicians  completed  the  tasks  more  quickly  on  the 
‘few’  pin  cannon  plugs  (1 1 .21  sec)  than  on  the  ‘many’  pin  cannon  plugs  (23.07  sec). 
However,  the  standard  deviations  were  too  large  (4.27  sec  and  14.59  sec  respectively)  to 
allow  an  initial  conclusion  to  be  drawn  on  the  credibility  of  the  hypothesis. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  8.  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  ‘few’  pin  cannon  plug  and  ‘many’ 
pin  cannon  plug  mean  enhanced  task  completion  times.  A  statistically  significant  result 
was  observed.  The  F-calculated  value  of  45.18  exceeded  the  F-critical  value  of  4.28  and 
thus  the  null  hypothesis  was  rejected.  The  two  cannon  plug  types  did  not  have  the  same 
mean  enhanced  task  completion  times. 

The  analysis  counters  the  hypothesis  that  there  would  be  no  difference  in  the 
enhanced  tasks  on  the  ‘few’  pin  cannon  plugs  and  the  ‘many’  pin  cannon  plugs. 


Table  8;  ‘Few’  vs.  ‘Many’  Pin  Cannon  Plug  Enhanced  Presentation  Method 

Mean  Task  Time  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Value 

Calculated 

F-Value 

P-Value 

Enhanced 

Error 

13018.547 

6627.828 

1 

23 

13018.547 

288.166 

4.280 

45.177 

0.000 
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Hypothesis  VI.  This  hypothesis  predicted  that  for  unenhanced  graphical 
presentations  only,  task  times  for  cannon  plugs  with  ‘few’  pins  (the  12  and  13  pin 
connectors)  would  be  shorter  than  task  times  for  cannon  plugs  with  ‘many’  pins  (the  55 
and  79  pin  connectors).  Technicians  completed  a  total  of  192  unenhanced  tasks  —  96 
utilizing  the  cannon  plugs  with  ‘few’  pins  and  96  utilizing  the  cannon  plugs  with  ‘many’ 
pins.  The  raw  data  showed  that  technicians  completed  the  tasks  more  quickly  on  the 
‘few’  pin  cannon  plugs  (13.25  sec)  than  on  the  ‘many’  pin  cannon  plugs  (39.54  sec). 
However,  the  standard  deviations  were  too  large  (5.43  sec  and  23.38  sec  respectively)  to 
allow  an  initial  conclusion  to  be  drawn  on  the  reliability  of  the  hypothesis. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  9.  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  ‘few’  pin  cannon  plug  and  ‘many’ 
pin  cannon  plug  mean  unenhanced  task  completion  times.  A  statistically  significant 
result  was  observed.  The  F-calculated  value  of  9.29  exceeded  the  F-critical  value  of  4.28 
and  thus  the  null  hypothesis  was  rejected.  The  two  cannon  plug  types  did  not  have  the 
same  mean  enhanced  task  completion  times. 

The  analysis  supports  the  hypothesis  that  the  unenhanced  ‘few’  pin  cannon  plug 
tasks  would  take  less  time  to  perform  than  the  unenhanced  ‘many’  pin  cannon  plug  tasks. 


Table  9:  ‘Few’  vs.  ‘Many’  Pin  Cannon  Plug  Unenhanced  Presentation  Method 

Mean  Task  Time  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Vaiue 

Calculated 

F-value 

P-Value 

Unenhanced 

Error 

200.083 

495.417 

1 

23 

200,083 

21.540 

4.280 

9.289 

0.006 
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Hypothesis  VII.  This  hypothesis  predicted  that  for  enhanced  graphical 
presentations  only,  there  would  be  no  difference  between  the  error  rates  of  tasks 
performed  with  cannon  plugs  with  ‘few’  pins  (the  12  and  13  pin  connectors)  and  cannon 
plugs  with  ‘many’  pins  (the  55  and  79  pin  connectors).  Technicians  completed  a  total  of 
192  enhanced  tasks  —  96  utilizing  the  cannon  plugs  with  ‘few’  pins  and  96  utilizing  the 
cannon  plugs  with  ‘many’  pins.  The  raw  data  showed  that  technicians  committed  5  errors 
when  they  performed  the  tasks  on  the  ‘few’  pin  cannon  plugs  and  9  errors  they  performed 
the  tasks  on  the  ‘many’  pin  cannon  plugs. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  10.  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  ‘few’  pin  cannon  plug  and  the 
‘many’  pin  cannon  plug  enhanced  task  completion  error  rates.  A  statistically  significant 
result  was  observed.  The  F-calculated  value  of  13.22  exceed  the  F-critical  value  of  4.28 
and  thus  the  null  hypothesis  was  rejected. 

The  analysis  counters  the  hypothesis  that  there  would  be  no  difference  between 
tasks  performed  with  cannon  plugs  with  ‘few’  pins  and  cannon  plugs  with  ‘many’  pins. 


Table  10:  ‘Few’  vs.  ‘Many’  Pin  Cannon  Plug  Enhanced  Presentation  Method 

Task  Error  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Value 

Calculated 

F-value 

P-Value 

Enhanced 

Error 

1.505 

2.62 

1 

23 

1.505 

0.114 

4.280 

13.215 

0.001 
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Hypothesis  VIII.  This  hypothesis  predicted  that  for  unenhanced  graphical 
presentations  only,  tasks  performed  with  cannon  plugs  with  ‘few’  pins  (the  12  and  13  pin 
connectors)  would  have  fewer  errors  than  tasks  performed  with  cannon  plugs  with  ‘many’ 
pins  (the  55  and  79  pin  connectors).  Technicians  completed  a  total  of  192  imenhanced 
tasks  —  96  utilizing  the  cannon  plugs  with  ‘few’  pins  and  96  utilizing  the  cannon  plugs 
with  ‘many’  pins.  The  raw  data  showed  that  technicians  committed  12  errors  when  they 
performed  the  tasks  on  the  ‘few’  pin  cannon  plugs  and  43  errors  they  performed  the  tasks 
on  the  ‘many’  pin  cannon  plugs. 

The  results  from  the  ANOVA  analysis  are  shown  in  Table  1 1 .  The  overall  F-test 
was  used  to  test  for  statistical  significance  at  the  0.05  significance  level.  The  null 
hypothesis  stated  there  was  no  difference  between  the  ‘few’  pin  cannon  plug  and  the 
‘many’  pin  cannon  plug  unenhanced  task  completion  error  rates.  A  statistically 
significant  result  was  observed.  The  F-calculated  value  of  62.69  exceeded  the  F-critical 
value  of  4.28  and  thus  the  null  hypothesis  was  rejected.  The  two  cannon  plug  types  did 
not  have  the  same  error  rate. 

The  initial  data  and  the  ANOVA  analysis  support  the  hypothesis  that  for 
unenhanced  presentations,  fewer  errors  would  be  committed  on  cannon  plugs  with  ‘few’ 
pins  than  on  cannon  plugs  with  ‘many’  pins. 


Table  11;  ‘Few’  vs.  ‘Many’  Pin  Cannon  Plug  Unenhanced  Presentation  Method 

Task  Error  Analysis 


Source  of 
Variance 

Sum  of 
Squares 

df 

Mean 

Square 

Critical 

F-Value 

Calculated 

F-value 

P-Value 

Unenhanced 

Error 

14.083 

5.167 

1 

23 

14.083 

0.225 

4.280 

62.694 

0.000 
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Summary  of  Results 

The  task  completion  time  and  error  rate  data  collected  for  this  thesis  indicate  that 
the  enhanced  graphical  presentation  method  fosters  improved  technician  performance 
over  the  unenhanced  graphical  presentation  method.  The  compiled  data  supports  the 
hypotheses  that  the  enhanced  presentations  reduce  technician  task  completion  time  and 
produce  fewer  technician  errors.  The  research  data  seems  to  indicate  that  the  enhanced 
presentations  produced  improved  technician  performance  in  every  measured  instance, 
except  one. 

It  was  hypothesized  that  the  enhanced  presentation  would  produce  equivalent  task 
completion  times  (Hypothesis  V)  for  the  ‘few’  and  ‘many’  pin  cannon  plugs.  The 
experiment  results  counter  Hypothesis  V  and  indicated  that  there  was  a  difference  in  the 
task  completion  time  between  the  different  cannon  plug  types.  The  ‘many’  pin  cannon 
plugs  took  longer  to  complete  the  experimental  tasks  than  the  ‘few’  pin  cannon  plugs. 

It  was  also  hypothesized  that  the  enhanced  presentation  would  produce  equivalent 
error  rates  (Hypothesis  VII)  for  the  ‘few’  and  ‘many’  pin  cannon  plugs.  The  experiment 
results  counter  Hypothesis  VII  and  indicated  that  there  was  a  difference  in  the  error  rate 
between  the  different  cannon  plug  types.  The  ‘many’  pin  cannon  plugs  statistically 
produced  more  errors  than  the  ‘few’  pin  cannon  plugs. 

Additionally,  when  the  overall  error  data  was  analyzed  for  differences  between 
the  ‘few’  and  ‘many’  pin  cannon  plugs  (Hypothesis  III),  no  statistically  significant 
difference  was  found.  These  outcomes  may  seem  to  indicate  that  technician  performance 
was  not  improved  over  the  unenhanced  presentation,  but  a  closer  examination  of  the  data 
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indicates  otherwise.  The  enhanced  presentation  resulted  in  lower  task  completion  times, 
fewer  errors,  and  smaller  standard  deviations  for  both  the  ‘few’  and  ‘many’  pin  cannon 
plugs  over  the  unenhanced  presentation  method.  The  enhanced  presentation  improved 
technician  performance  for  both  cannon  plug  types  even  though  it  did  not  eliminate  the 
difference  between  them. 

The  qualitative  data  collected  from  the  test  participants  was  used  primarily  by 
Armstrong  Laboratory  to  evaluate  the  HMD  system  that  they  provided  for  this  thesis 
effort.  The  questions  and  replies  dealt  with  the  usability  of  the  HMD  system  and  are 
discussed  in  the  next  chapter. 
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V.  Discussion,  Conclusions,  and  Recommendations 


Overview 

This  chapter  contains  a  discussion  of  the  results  of  the  quantitative  data  collected 
during  this  research  and  its  support  for  the  experimental  hypotheses  that  were  proposed. 
The  qualitative  data  that  was  collected  is  addressed  and  the  important  findings  are 
highlighted.  Conclusions  are  drawn  from  the  results  of  this  experiment  and  analysis  and 
recommendations  for  further  research  are  proposed. 

Discussion  of  Quantitative  Results 

The  overall  mean  task  completion  time  and  task  error  data  collected  for  this  thesis 
supports  the  hypotheses  that  when  the  technicians  used  the  enhanced  data  presentation  to 
complete  the  tasks  they  performed  better.  The  mean  task  completion  time  analysis  of  the 
measured  instances:  enhanced  vs.  unenhanced  overall,  ‘few’  vs.  ‘many’  pin  coimectors 
overall,  and  ‘few’  vs.  ‘many’  pin  connectors  (unenhanced  presentation  only)  indicated 
that  the  proposed  hypotheses  where  indeed  correct.  The  task  error  rate  analysis  of  the 
measured  instances:  enhanced  vs.  unenhanced  overall  and  ‘few’  vs.  ‘many’  pin 
connectors  (unenhanced  presentation  only)  also  indicated  that  the  proposed  hypotheses 
where  indeed  correct. 

It  was  initially  believed  that  the  enhanced  presentation  would  produce  almost 
equal  error  rates  and  mean  task  times  between  the  two  cannon  plug  groups  (‘few’  and 
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‘many’  pins),  but  the  collected  data  indicates  otherwise.  The  mean  task  completion  time 
for  the  ‘few’  vs.  ‘many’  pin  connector  was  shown  to  be  different,  countering  the  proposed 
hypothesis.  The  analysis  for  this  hypothesis  indicated  that  the  ‘many’  pin  connector 
mean  task  time  was  longer  than  the  ‘few’  pin  connector.  The  task  error  rate  for  the  ‘few’ 
vs.  ‘many’  pin  connector  was  also  shown  to  be  different.  Fewer  errors  were  committed 
when  the  technicians  used  the  ‘few’  pin  connector  than  the  ‘many’  pin  cormector. 

The  data  for  the  enhanced  presentation  method  may  seem  to  indicate  that 
technician  performance  was  not  improved  over  the  unenhanced  presentation,  but  a  closer 
examination  of  the  data  indicates  otherwise.  The  enhanced  presentation  resulted  in  lower 
task  completion  times,  fewer  errors,  and  smaller  standard  deviations  for  both  the  ‘few’ 
and  ‘many’  pin  cannon  plugs  over  the  unenhanced  presentation  method.  The  enhanced 
presentation  improved  technician  performance  for  both  cannon  plug  types  even  though  it 
did  not  eliminate  the  difference  between  them. 

The  overall  results  of  this  research  indicate  that  using  enhanced  graphical 
presentation  methods  on  an  HMD  produced  better  technician  performance,  as  measured 
in  mean  task  completion  time  and  error  rate.  A  closer  analysis  of  the  collected  data  does 
raise  a  few  unanswered  concerns  that  were  not  addressed  in  the  initial  experimental 
design.  These  concerns  are  detailed  below. 

Cannon  Plug  Design.  Besides  the  indexing  scheme,  the  cannon  plugs  were  also 
different  physical  sizes,  physically  arranged  differently  (see  Figures  5  and  6),  used 
different  size  connector  pins,  had  different  indexing  number  colors,  and  had  different 
color  cannon  plug  faces.  These  variables  were  not  controlled  for,  nor  were  they 
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considered  important  in  the  experimental  design.  Because  of  the  appearance  of  the 
cannon  plug  layout  and  learning  effect  discussed  later  in  this  chapter,  these  variables  may 
need  to  be  reinvestigated. 

Cannon  Plug  Layout.  The  cannon  plugs  used  in  this  thesis  had  both  alphabetical 
and  numerical  indexing  schemes.  The  cannon  plugs  also  were  indexed  in  a  variety  of 
different  ways.  These  variables  were  not  considered  important  enough  to  control  for  in 
the  experimental  design.  Looking  at  the  raw  data,  as  shown  in  Table  12,  these  variables 
appear  to  produce  some  varying  results.  It  appears  that  the  indexing  scheme  produces 
different  results  between  the  ‘few’  pin  connector  (12  and  13)  and  the  ‘many’  pin 
connector  (55  and  79)  groups. 


Table  12:  Raw  Cannon  Plug  Type  Data 


Cannon  Plug 
Type 

Indexing 

Scheme 

Mean  Task 
Time  (sec) 

Error 

Rate 

■aiBi 

Alphabetical 

13.32 

3 

Numerical 

11.14 

14 

Alphabetical 

30.67 

35 

■BH 

Numerical 

31.95 

17 

It  could  be  that  the  indexing  scheme,  marking  style,  character  size,  color,  or  type 
on  each  of  the  cannon  plugs  produces  different  results.  The  alphabetical  indexing  may  be 
more  difficult  to  read  on  the  12  pin  cannon  plug  than  the  numerical  indexing  on  the  13 
pin  cannon  plug;  but,  as  seen  in  the  collected  data,  produces  less  errors.  As  \vitnessed  in 
the  quicker  task  completion  time  and  higher  error  rate,  the  indexing  on  the  55  pin  cannon 
plug  may  be  easy  to  read,  but  more  obscure  than  the  79  pin  cannon  plug.  Because  the 
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experimental  design  was  not  set  up  to  test  this  relationship,  it  can  only  be  said  that  the 
appearance  of  an  effect  exists  and  is  discussed  later  in  the  recommendations  section  of 
this  chapter. 

Learning  Effect.  The  randomized  block  design  of  the  HMD  graphical 
presentation  order  was  used  to  minimize  the  learning  effect  from  occurring.  As  shown  in 
Table  13,  it  appears  that  the  learning  effect  does  indeed  occur.  The  data  is  arranged  in 
groups  of  four  trials.  The  16  trials  each  technician  performed  were  divided  into  blocks  of 
four  trials  and  tallied  across  all  24  applicable  data  sets.  It  appears  that  the  randomized 
block  design  of  the  HMD  graphical  presentation  order  did  produce  a  small  order  effect. 
The  second  set  of  eight  trials  appears  to  have  a  lower  mean  task  time  and  error  rate. 


Table  13:  Raw  Trial  Group  Data 


Data 

Group 

Mean  Task 
Time  (sec) 

Error 

Rate 

Trials  1-4 

24.51 

20 

Trials  5-8 

24.82 

18 

Trials  9-12 

19.25 

15 

Trials  13-16 

18.47 

16 

It  appears  that  technicians  experienced  the  learning  effect  after  performing  the 
first  set  of  eight  trials.  It  is  likely  that  after  seeing  all  eight  presentation  types  in  the  first 
eight  trials,  the  technicians  learned  the  presentation  scheme.  It  could  also  be  that  the 
training  session  was  not  long  enough  and  technicians  simply  took  a  while  to  learn  the 
HMD  setup  and  display  information.  Once  again,  because  the  experimental  design  was 


54 


not  set  up  to  test  this  relationship,  it  can  only  be  said  that  the  appearance  of  an  effect 
exists. 

Discussion  of  Qualitative  Results 

Data,  in  the  form  of  a  questionnaire  and  interview,  were  collected  from  all  28  of 
the  test  participants.  Most  of  the  data  was  directed  at  the  usability  and  comfort  of  use  of 
the  HMD.  This  data  was  used  by  Armstrong  Laboratory  in  a  separate  study  involving 
different  wearable  computer  systems.  The  data  that  did  pertain  to  this  study  is  chronicled 
below. 

Information  Presented  on  the  HMD.  The  technicians  were  asked  if  the 
information  presented  on  the  HMD  was  readable  and  if  and  when  they  noticed  a 
difference  in  presentation  types  (unenhanced  vs.  enhanced).  All  28  technicians  indicated 
that  the  information  itself  was  mostly  legible.  Some  references,  such  as  numbers  and  the 
extreme  comers  of  the  HMD  were  said  to  be  fuzzy;  but,  all  indicated  that  the  HMD  was 
adequate  in  presenting  the  data. 

Another  comment  of  importance  to  note  was  that  some  technicians  were 
apprehensive  about  relying  on  the  enhanced  presentations.  The  reasons  given  for  not 
wanting  to  use  the  enhancements  were  that  they  did  not  tmst  the  presentation,  the  plugs 
were  easy  to  read,  and  they  felt  more  comfortable  counting  the  pins  out.  The  technicians 
indicated  that  the  pin  counting  and  mistmst  of  presentation  were  the  result  of 
encountering  numerous  technical  orders  that  were  improperly  marked  and  that  they  did 
not  tmst  technical  data. 
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HMD  Use.  The  technicians  in  this  study  only  wore  the  HMD  system  for  about  a 
half  an  hour.  The  tasks  that  they  performed  were  in  a  static  environment  with  few  outside 
distracters  or  interferences.  Even  with  this  limited  experience,  the  technicians  provided 
insightful  comments  about  the  HMD  and  its  possible  uses. 

Technicians  were  asked  about  the  HMD  and  their  ability  to  use  it  in  their  present 
jobs.  The  HMD  setup  used  in  this  research  posed  many  concerns  to  the  technicians.  The 
major  concerns  were  the  eye  strain  brought  on  by  switching  between  the  cannon  plug  and 
HMD,  the  extra  clips  and  accessories  that  interfered  with  the  HMD  use  (there  was  an 
earpiece  and  microphone  that  were  not  used),  and  the  weight  of  the  system  causing 
headaches  and  sweating  that  would  cause  the  HMD  to  slip  down  on  the  forehead. 

However,  the  technicians  overwhelmingly  indicated  that  the  HMD  would  be  an 
asset  to  them  in  the  performance  of  their  daily  duties.  Twenty-three  of  the  technicians 
said  the  HMD  could  be  used  in  troubleshooting,  schematic  tracing,  or  tasks  that  required 
their  hands  to  be  free.  Other  suggested  uses  were  in  dark,  tight  places,  during  nighttime 
operations,  and  on  tasks  that  required  two  people  to  perform.  The  suggestion  was  that 
technicians  could  be  electronically  connected  to  the  same  reference  and  could  work 
together  better  by  using  the  HMD. 

Technician  Suggestions.  The  techniciems  were  asked  if  they  could  think  of  any 
tasks  they  perform  or  maintenance  scenarios  in  which  an  HMD  setup  might  be  useful. 
Many  suggestions  were  provided  and  many  were  extremely  perceptive.  The  most 
mentioned  advantage  would  be  from  putting  Technical  Orders  (TOs)  on  the  HMD. 
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Technicians  felt  that  having  wiring  diagrams,  schematics,  and  check  lists  on  an 
HMD  would  enable  technicians  to  concentrate  on  the  task  and  not  the  data  needed  to 
perform  the  task.  As  one  technician  put  it,  some  tasks  require  one  hand  to  hold  a 
flashlight,  one  hand  to  hold  the  test  leads,  one  hand  to  hold  the  screwdriver,  one  hand  to 
operate  the  test  equipment,  and  one  hand  to  turn  the  page  in  the  TO  --  “even  an  octopus 
would  be  hard  pressed”  to  do  the  task  properly. 

Other  suggestions  included  hooking  the  HMD  up  to  an  on-line  system  that  would 
enable  the  wearer  to  access  any  and  all  the  information  needed  to  do  their  job.  Suggested 
information  included  supply  information,  core  automated  maintenance  system  data,  all 
the  aircraft  maintenance  history,  the  world-wide  web,  and  e-mail.  Incorporating  the 
HMD  into  a  complete  system  of  miner’s  light,  ear  defenders,  radio,  and,  inter-phone  was 
also  frequently  mentioned. 

Conclusions 

The  results  of  this  research  suggest  that  enhanced  cannon  plug  pin  selection  task 
data  presented  on  an  HMD  would  be  useful  in  improving  maintenance  technician 
performance.  Technicians  completed  the  cannon  plug  pin  selection  and  connection  tasks 
quicker  and  committed  fewer  errors  when  they  used  the  enhanced  HMD  data  presentation 
to  perform  the  task.  Analysis  of  the  obtained  data  revealed  statistically  significant  results 
that  indicated  that  the  enhanced  data  presentation  method  did  reduce  the  task  performance 
time  and  error  rate. 
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Qualitative  data  obtained  from  technician  feedback  also  suggest  that  HMD 
systems  would  be  advantageous  for  and  used  by  technicians  in  the  performance  of  their 
maintenance  duties.  Technicians  reported  that  they  had  very  few  problems  using  the 
HMD  and  would  like  to  see  some  type  of  HMD  system  available  for  their  use. 

Recommendations  for  Future  Research 

The  full  portability  capabilities  of  current  HMD  systems  was  not  tested  in  this 
study.  The  technicians  that  participated  in  this  study  did  not  have  to  move  around  to 
perform  the  required  maintenance  tasks.  Flightline  maintenance  tasks  and  tasks  that 
require  technicians  to  move  around  and  interact  with  their  environment  should  be 
investigated.  The  technicians  in  this  study  suggested  system  troubleshooting  tasks, 
inspection  tasks,  schematic  and  troubleshooting  chart  reading  tasks,  and  tasks  that 
required  coordinated  teams  of  technicians  operating  concurrently  as  possible  future 
avenues  of  study. 

HMDs  and  HMD  technology  has  greatly  advanced  in  just  the  past  few  years. 
According  to  three  articles  from  inquiry.com  on  the  World  Wide  Web  (Displays,  1996; 
Little  Displays,  1996;  Motorola,  1995),  newer  systems  are  incorporated  into  regular 
eyeglass  frames,  offer  look-through  capability,  and  are  completely  portable.  These 
systems  could  eliminate  the  problems  of  the  current  HMD  system  cited  by  the  technicians 
used  in  this  study.  Investigation  of  such  HMD  systems  to  see  if  they  could  be  used  in  a 
maintenance  environment  may  result  in  improved  technician  performance. 
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The  impetus  for  this  study,  cannon  plug  pin  selection  and  connection  errors,  is 
still  not  completely  addressed  in  this  thesis.  Future  studies  should  investigate  various 
ways  to  further  reduce  technician  errors.  Other  HMD  types  and  improvements  to  cannon 
plug  design,  such  as  pin  layout,  number  of  pins,  pin  layout  style,  color  and  numbering 
schemes,  and  pin  indexing  (alphabetical  vs.  numerical)  should  be  studied  to  see  if  they 
could  improve  technician  performance.  Additionally,  a  more  rigorous  examination  in 
graphical  enhancements  should  be  performed. 

Technicians  should  be  thoroughly  familiar  with  the  graphical  enhancements  and 
how  to  use  them.  The  technicians  should  then  be  instructed  to  trust  and  use  the 
enhancements.  Identifying  the  enhancements  and  forcing  the  technicians  to  use  them 
would  eliminate  the  uncertainty  of  whether  or  not  the  technicians  actually  noticed  the 
enhancements  and  if  they  actually  used  them,  as  seen  in  this  study.  Identifying  improved 
methods  to  present  task  information  and  the  most  user  fnendly  cannon  plug  design  could 
help  to  improve  technician  performance  resulting  in  reduced  maintenance  costs  and 
increased  equipment  availability.  For  these  reasons,  further  research  in  the  presentation 
of  task  information  and  cannon  plug  design  should  be  pursued. 
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Appendix  A:  User  Evaluation  Questionnaire 


Test  Subject  Number: _ 

Please  answer  the  following  questions  based  on  your  participation  in  the 
experiment.  This  questiormaire  is  divided  into  three  sections,  with  questions  on  visual 
aspects  related  to  the  use  of  the  HMD  display,  questions  on  the  information  presented  on 
the  HMD,  and  general  questions  on  the  HMD  usability.  Please  read  the  questions 
carefully  and  put  a  check  mark  in  the  appropriate  block.  The  scale  for  the  first  two 


sections  is  as  follows: 

Very'  Satisfactory 

Very  Unsatisfactory 

A 

B 

C 

D  E 

I.  Questions  on  the  visual  aspects  of  the  display: 


~~cr~ 

~rr“ 

1. 

Capability  of  switching  your  attention  from  the 
display  to  your  work 

m 

■ 

m 

■ 

Adequacy  of  the  screen  size  tor  displayed  information 

3. 

Head  piece  comfort 

4. 

Ability  to  focus  the  HMD 

5. 

Brightness  of  the  HMD  display 

6. 

Glare  on  the  HMD  screen 

7. 

Capability  of  positioning  the  HMD 

II. 

Questions  on  the  information  oresented: 

A 

B 

mm 

D 

E 

1. 

Readability  of  all  of  the  information  on  the  screen 

T' 

Spacing  of  information  on  the  display  screen 
(lack  of  clutter,  etc.) 

■ 

■ 

■ 

■ 

Cannon  plug  picture  graphics 

d 

Resolution  and  clarity  of  the  graphic  displays 

5. 

Contrast  between  the  information  displayed  and 
the  background 

■ 

m 
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Test  Subject  Number: _ 

Please  answer  the  following  questions  according  to  your  experience  with  the 
HMD  in  this  experiment.  The  scale  for  the  this  section  is  as  follows: 


Not  a  Problem 

Serious  Problem 

A 

B 

C 

D 

E 

III.  General  questions  on  the  HMD  use: 


■91 

wm 

u 

Eye  strain 

a 

Blurring 

3 

After/ghost  images  after  discontinued  use  of  HMD 

El 

Headaches  from  using  the  HMD 

Thank  you  for  participating  in  this  experiment. 


Appendix  B:  Structured  Interview 


Test  Subject  Number: _ 

This  interview  is  conducted  after  the  test  subject  has  completed  the  experiment 
and  has  filled  out  the  User  Evaluation  Questioimaire,  Appendix  A.  The  interview  is 
conducted  in  the  same  room  in  which  the  technician  performed  the  tasks.  The  test 
administrator  asks  the  following  questions  and  solicits  responses  from  the  test  subject. 
Printed  copies  of  the  HMD  presented  data,  Appendix  C,  are  used  as  references  while 
these  questions  are  being  asked  and  answered. 

The  following  questions  pertain  to  the  information  presented  on  the  display  device: 


1 .  Which  type  of  information  (text,  graphics)  was  easier  to  read  on  the  HMD?  Why? 

2.  Were  the  graphics  detailed  enough  so  that  you  could  recognize  the  features  and  index 
of  the  carmon  plug  being  represented?  How? 

3.  Did  any  of  the  presentation  methods  enable  you  to  better  identify  and  locate  the  pins 
that  you  were  to  connect?  WTiich  ones?  How?  (Use  the  printed  screen  graphics 
while  asking  this  question.) 

4.  In  your  own  words,  describe  the  difference  between  the  different  presentation 
methods. 

5.  WTien  presented  with  the  unenhanced  data,  did  you  use  the  HMD  or  did  you  just  use 
the  cannon  plug  itself? 

6.  When  presented  with  the  smaller  numbered  cannon  plugs,  did  you  use  the  HMD  or 
just  the  cannon  plugs  themselves? 
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Test  Subject  Number: _ 

The  following  questions  pertain  to  the  HMD  used  during  the  evaluation: 

1 .  How  difficult  was  it  to  adapt  to  the  HMD?  Explain. 

2.  Can  you  describe  anything  you  did  not  like  about  the  HMD? 

3.  Did  you  have  any  problems  with  glare  on  the  HMD?  If  so,  how  did  you  compensate? 

4.  Would  this  device  work  on  the  flightline?  Why  or  why  not? 

5.  Which  maintenance  tasks  would  be  easier  if  technical  data  was  presented  on  the 
HMD? 

The  following  questions  pertain  to  the  physical  aspects  of  using  the  HMD: 

1 .  Did  you  have  any  visual  problems  (blurring,  problems  focusing,  flickering,  etc.)  while 
using  the  HMD? 

2.  Did  you  get  a  headache  or  suffer  from  eye  strain  while  using  the  HMD? 

3.  Did  you  have  any  problems  with  the  HMD  headband? 

4.  (If  the  technician  wears  glasses)  was  it  difficult  using  the  HMD  while  wearing 
glasses? 

The  following  questions  pertain  to  the  cannon  plug  pin  selection/connection  task 
performed  during  the  evaluation: 

1 .  Can  you  think  of  any  maintenance  tasks  in  which  the  HMD  would  work? 

2.  Are  there  any  physically  distinguishable  tasks  where  using  the  HMD  would  work 
particularly  well  (tasks  requiring  hands  free  or  tight  places)?  Not  work? 
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Appendix  C:  HMD  Presented  Data 


Practice  Cannon  Plug 

The  following  are  pictures  of  the  actual  screens  that  are  displayed  on  the  HMD  for 
the  technician  during  the  experiment.  The  unenhanced  screens  have  the  pin  numbers  to 
connect  in  the  “connect  from  -  to”  table  to  the  left  of  the  cannon  plug  picture.  The 
unenhanced  presentations  have  the  same  pin  pairs  as  the  enhanced  presentations. 


51  Contacts 


Connect 

From 

To 

Figure  10:  Practice  Cannon  Plug  HMD  Graphical  Presentation 


The  practice  tasks  show  the  same  graphical  presentation  for  each  of  the  four  tasks. 
The  only  difference  is  that  the  pins  in  the  “connect  to  from  -  to”  table  were  changed  for 
each  of  the  four  practice  tasks. 
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12  Pin  Cannon  Plug 


Only  the  basic  screen  is  shown  for  the  unenhanced  graphical  presentations.  The 
four  unenhanced  graphical  presentations  use  the  same  pin  combinations  as  the  enhanced 
graphical  presentations. 

Unenhanced 
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Connect 

From 

To 

Enhanced 
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1  Connect  1 

From 

To 
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K 

12  CONTACTS 
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Connect 
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To 

D 

H 
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To 

D 

J 
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To 

E 

K 
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Figure  11:  12  Pin  Cannon  Plug  HMD  Graphical  Presentations 
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13  Pin  Cannon  Plug 


Only  the  basic  screen  is  shown  for  the  unenhanced  graphical  presentations.  The 
four  unenhanced  graphical  presentations  use  the  same  pin  combinations  as  the  enhanced 
graphical  presentations. 
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Figure  12:  13  Pin  Cannon  Plug  HMD  Graphical  Presentations 
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55  Pin  Cannon  Plug 


Only  the  basic  screen  is  shown  for  the  imenhanced  graphical  presentations.  The 
four  unenhanced  graphical  presentations  use  the  same  pin  combinations  as  the  enhanced 
graphical  presentations. 
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Figure  13:  55  Pin  Carmon  Plug  HMD  Graphical  Presentations 
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79  Pin  Cannon  Plug 


Only  the  basic  screen  is  shown  for  the  unenhanced  graphical  presentations.  The 
four  unenhanced  graphical  presentations  use  the  same  pin  combinations  as  the  enhanced 
graphical  presentations. 
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Figure  14:  79  Pin  Cannon  Plug  HMD  Graphical  Presentations 
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Appendix  D:  Experimental  Plan 


(Adapted  from  Masquelier,  1991) 

This  appendix  outlines  the  procedures  that  are  used  in  this  experiment  and  an 
overview  of  the  experiment  background.  Chapter  Three  contains  a  complete  description 
of  the  experiment  methodology  used  in  this  research 


Description  of  Evaluation 


Purpose 

The  purpose  of  this  study  is  to  evaluate  the  effect  that  information  presented  on  a 
monocular  Head-Mounted  Display  (HMD)  has  on  technician  performance  of  cannon  plug 
pin  selection  and  connection  tasks. 

Subjects 

Avionics  maintenance  technicians  from  the  2d  Bomb  Wing  at  Barksdale  AFB,  LA 
are  test  subjects  in  this  experiment.  The  technicians  are  volunteers  randomly  chosen  from 
the  pool  of  available  personnel  at  the  base.  Criteria  for  consideration  in  the  available  pool 
are  based  on  time  in  service,  time  on  the  flightline/bench,  current  duties,  and  eyesight. 
Technicians  had  to  have  a  minimum  of  four  years  continuous  active  duty  experience, 
must  have  been  working  on  the  flightline/bench  for  at  least  two  years,  and  must  currently 
be  performing  maintenance  duties  on  the  flightline/bench.  Technicians  must  also  have  at 
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least  20/20  corrected  vision  and  cannot  wear  bifocals  or  trifocals  in  order  to  be  considered 
in  the  available  pool  of  technicians  for  this  study. 

Hardware 

There  are  several  pieces  of  equipment  used  in  this  research.  The  HMD  is  a 
monocular  display  attached  to  a  headband  and  is  made  by  Kopin™.  A  portable  laptop 
computer  is  used  by  the  test  administrator  as  the  CPU,  memory,  and  control  device. 

The  test  apparatus  are  five  multi-pin  cannon  plug/breakout  box  assemblies.  The 
four  assemblies  used  for  the  experiment  are  circular.  The  fifth  assembly,  which  is  used 
for  practice,  orientation,  and  training,  is  rectangular. 

Software 

The  software  used  in  the  experiment  is  a  locally  developed  test  program  in  Visual 
Basic™  4.0  language.  The  technicians  are  presented  a  menu  driven  test  program  that 
provides  visual  cues  to  follow.  The  test  administrator  steps  through  the  program  using  a 
laptop  computer  as  the  technician  indicates  he  or  she  is  ready  to  continue. 

Experiment  Tasks 

Each  technician  performs  sixteen  measured  caimon  plug  pin  selection/connection 
tasks  wearing  the  HMD.  Eight  of  the  tasks  are  completed  using  the  graphically  enhanced 
presentation  and  eight  are  completed  using  the  unenhanced  presentation.  The  tasks  are 
simple  by  design:  The  technicians  insert  test  adapter  connectors  in  two  specific  cannon 
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plug  pins.  The  presentation  on  the  HMD  specifies  which  connector  to  use  and  which  pins 
to  connect.  Table  12  shows  what  connector  type  and  pins  are  used  for  each  of  the  sixteen 
test  conditions. 


Table  14:  Task  Condition  Identification 


1  Task  Conditions 

Connector  Type 
(#  of  Pins) 

Pins  to  Connect  | 

Enhanced 

Unenhanced 

First 

Second 

E9a 

U9a 

79 

24 

46 

E5a 

U5a 

55 

N 

s 

E3a 

U3a 

13 

9 

10 

E2a 

U2a 

12 

A 

F 

E9b 

U9b 

79 

15 

59 

E5b 

U5b 

55 

F 

X 

E3b 

U3b 

13 

4 

12 

E2b 

U2b 

12 

D 

J 

E9c 

U9c 

79 

6 

54 

E5c 

U5c 

55 

C 

V 

E3c 

U3c 

13 

2 

11 

E2c 

U2c 

12 

E 

K 

E9d 

U9d 

79 

47 

67 

E5d 

USd 

55 

S 

h 

E3d 

U3d 

13 

5 

6 

E2d 

U2d 

12 

D 

H 

Note:  These  cannon  plugs  are  circular.  The  cannon  plugs  differ  by 
the  number  of  connector  pins  they  have  and  their  pin  layout. 


Conditions 


Table  1 5  shows  the  test  subject  number  and  the  task  condition  order  the  technician 
performs  during  each  of  the  sixteen  tasks.  The  28  test  subjects  are  shown.  The  design  of 
the  task  conditions  in  Table  15  is  accomplished  by  using  block  randomization.  This  is 
done  to  avoid  data  presentation  order  effects.  Chapter  Three  provides  further  clarification 
on  the  experiment  research  methodology. 
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Table  15:  Task  Condition  Order 


Subject 

Number 

1  T£sk  Number  | 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

IS 

14 

15 

16 

1 

E9c 

E5c 

U2c 

E3d 

E2a 

U5d 

U9a 

U3b 

U9b 

U2d 

ESc 

E2b 

USa 

Bd 

Ba 

E5c 

E3c 

E2d 

L»a 

E9c 

U5b 

U2a 

U3d 

E9d 

E2b 

LBd 

ESb 

USa 

ESa 

U9b 

U2c 

E9d 

E2d 

E3b 

U9b 

U2c 

ESd 

USa 

E2b 

L»a 

ESa 

ESc 

U2a 

E9c 

U9b 

Wc 

U2} 

E5c 

E3b 

E2c 

E9c 

USd 

ESb 

Bd 

E2d 

E9a 

ESa 

U2a 

E3c 

U9a 

E5b 

E2b 

U3b 

U5a 

E9d 

E2c 

E9c 

U9b 

ESa 

U2d 

ESc 

U3c 

U2b 

U9b 

E5c 

E3a 

E2a 

Ba 

USb 

ESd 

E2d 

E9d 

USd 

U9c 

ESb 

U2c 

lea 

E5a 

E2a 

E3b 

E9c 

U5b 

U2c 

U3c 

U9d 

E9b 

use 

E2d 

ESc 

U9a 

U2b 

ted 

ESd 

8 

U2b 

E3a 

e9d 

U3d 

U9a 

ESb 

E2d 

ESd 

E9b 

E2a 

ESb 

U9c 

USc 

Lee 

U2c 

9 

E2b 

E3b 

E9b 

U3d 

Ute 

ESa 

U2d 

U2a 

E9d 

ESa 

USd 

USc 

U9c 

ESb 

E2c 

10 

U5a 

E2a 

E9d 

U2c 

U9a 

ESb 

ESa 

ESC 

U2d 

U9b 

USd 

E2b 

E9c 

Bd 

11 

E2c 

e% 

U3d 

use 

U2d 

E3b 

U9d 

Bb 

ESd 

U2b 

ESC 

E9a 

U9c 

tea 

E2a 

12 

E5c 

E9b 

U2d 

E3d 

E2b 

U5a 

U9d 

USb 

ESa 

U9a 

E2a 

E9c 

USc 

U2c 

ESd 

13 

U5d 

U2a 

E9d 

E3b 

E5c 

U3d 

E2d 

U9c 

USb 

E9b 

E2b 

USc 

ESa 

ESa 

U2c 

L^ 

14 

U2b 

E9b 

E3c 

E2a 

U9d 

ESd 

U3a 

E9c 

USd 

U2d 

E2c 

ESb 

U9a 

E% 

15 

U2c 

U3d 

U9c 

E2a 

E9d 

E3a 

ESb 

E2d 

ESC 

E9b 

ESd 

U2b 

USb 

Ba 

lec 

16 

U5c 

U3c 

U2c 

U9b 

E2a 

E3a 

e9a 

ESb 

ESd 

U2b 

E9d 

USd 

E2d 

ESb 

tea 

U9c 

17 

U2c 

U5b 

E9b 

U3a 

E5a 

E3d 

U9c 

E2b 

ESC 

U2d 

U9d 

USc 

E9a 

Led 

ESb 

E2a 

18 

U9c 

E5a 

E3b 

E2a 

U2c 

U3a 

e9a 

USb 

U2b 

USd 

U9b 

USc 

E2d 

ESc 

E9d 

ESd 

19 

E9d 

U2c 

U3c 

E5d 

E2a 

U9b 

ESd 

USa 

USa 

E2d 

U9a 

U2b 

E9c 

ESb 

ESc 

20 

E2a 

E5d 

E9d 

U3d 

E3a 

U2b 

U9c 

USa 

USb 

E9b 

E2d 

ESc 

USb 

U2c 

ESc 

U9a 

21 

E2c 

E3d 

use 

E9d 

U3b 

U2a 

ESb 

ESd 

ESa 

U2d 

U9b 

USc 

E2b 

E9c 

USc 

22 

E9b 

U2b 

U3c 

E5b 

E3a 

E2c 

USa 

U9c 

USb 

ESC 

U2a 

E2d 

USd 

E9a 

ESd 

23 

U9d 

E5c 

E2b 

E3d 

U5b 

U2a 

E9b 

U3c 

U9c 

USb 

U2c 

ESd 

ESa 

E9a 

E2d 

24 

U2d 

E5a 

U3a 

E9d 

U9c 

E2c 

E3c 

use 

E2a 

U9a 

USb 

ESd 

USb 

ESd 

U2b 

Bb 

25 

E5b 

E9a 

U3d 

U2d 

U5d 

U9b 

E2b 

ESb 

U9c 

use 

use 

E2c 

E9d 

ESa 

U2a 

ESa 

26 

E2b 

E3d 

U9d 

E5c 

U2d 

U3a 

Be 

E2c 

ESC 

Bb 

ESb 

USb 

U9a 

U2a 

27 

U2d 

E5c 

U3d 

E9b 

E3a 

U5b 

E2b 

U9d 

Be 

USb 

ESa 

U2c 

U9a 

E2a 

USd 

ESc 

28 

U2a 

E3a 

E9d 

use 

E5a 

U9c 

E2c 

USd 

ESC 

E9a 

E2d 

USd 

U2b 

ESb 

USb 

U9b 

Note:  Avionics  technicians  completed  16  tasks  each,  eight  graphically  enhanced  and  eight  unenhanced. 
Experiment  Hypotheses 


1 .  Fewer  cannon  plug  pin  connection  errors  are  made  using  the  graphically 
enhanced  presentation  on  the  HMD  than  the  unenhanced  presentation. 

2.  Pin  selection/connection  takes  less  time  using  the  graphically  enhanced 
presentation  method  than  the  unenhanced  presentation. 

3.  Cannon  plugs  with  fewer  pins  will  have  fewer  connection  errors  than  cannon 
plugs  with  many  pins. 
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4.  Cannon  plugs  with  fewer  pins  will  take  less  time  to  complete  the  task  than 
cannon  plugs  with  many  pins. 

5.  For  enhanced  presentations,  there  is  no  statistically  significant  difference  in 
time  to  complete  the  tasks  between  cannon  plugs  with  many  and  few  pins. 

6.  For  unenhanced  presentations,  caimon  plugs  with  few  pins  will  take  less  time 
to  complete  the  tasks  than  cannon  plugs  with  many  pins. 

7.  For  enhanced  presentations,  there  is  no  statistically  significant  difference  in 
the  number  of  task  completion  errors  between  cannon  plugs  with  many  and 
few  pins. 

8.  For  unenhanced  presentations,  cannon  plugs  with  few  pins  will  have  less 
errors  in  task  completion  than  cannon  plugs  with  many  pins. 


Data  Collected 


The  total  time  to  complete  each  task  and  the  cannon  plug  pin  selection/connection 
error  rate  is  collected.  Additionally,  demographic  data  is  collected  using  a  Personal 
Background  Form  (Appendix  E).  During  the  experiment,  notes  and  observations  are 
documented  (Appendix  G).  After  the  technician  completes  the  experiment,  the  User 
Evaluation  Questionnaire  is  used  to  ascertain  the  users’  evaluation  of  the  experiment 
(Appendix  A).  A  structured  interview  is  then  performed  to  receive  comments  on  the 
HMD,  presentation  methods,  and  the  techniques  used  for  pin  identification  (Appendix  B). 
Determining  users’  likes,  dislikes,  and  concerns  is  important  in  trying  to  design  a  system 
that  meets  the  users’  needs. 
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Controls 


The  following  actions  are  performed  to  control  for  experiment  variation: 

1 .  Test  subjects  are  avionics  technicians  and  have  a  minimum  of  four  years 
active  duty  Air  Force  time  in  service,  two  years  flightline/bench  experience, 
and  their  current  job  is  on  the  flightline/bench.  This  provides  a  common 
experience  minimum  for  the  tests  and  allowed  experience  levels  to  be 
controlled  for. 

2.  Given  #1 ,  test  subjects  are  volunteers  randomly  selected  from  the  available 
pool  of  avionics  technicians  at  the  2d  Bomb  Wing,  Barksdale  AFB,  LA. 

3.  The  same  experimenter  conducted  the  data  collection  activities. 

4.  The  graphical  data  presented  in  the  enhanced  and  unenhanced  tasks  is  the 
same  for  all  test  subjects.  The  order  of  presentation  is  counterbalanced  to 
account  for  any  presentation  order  anomalies. 

5.  Test  subjects  have  at  least  20/20  corrected  vision  and  do  not  wear  bifocals  or 
trifocals. 

6.  The  task  performance  time  is  computed  by  the  computer  program  itself  The 
test  administrator  starts  and  stops  the  test  sequence  at  the  direction  of  the  test 
subject  and  the  program  automatically  computes  the  test  duration  time. 

7.  The  test  subjects  are  allowed  to  orient  themselves  on  the  task  to  be  performed 
before  beginning  the  test  (cannon  plug  pin  layout  and  which  test  adapters  to 
use).  This  is  done  to  provide  a  common  starting  point  for  the  tasks. 


Experiment  Procedures 


Task  Assignment 


Avionics  technicians  act  as  test  subjects.  The  technicians  are  volunteers  randomly 
chosen  from  the  available  pool  and  assigned  one  of  the  subject  numbers  outlined  in  Table 
15.  Each  technician  performs  16  cannon  plug  pin  selection/connection  tasks.  Eight  of 
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those  tasks  are  done  using  the  HMD  and  an  enhanced  graphical  presentation  of  the 
cannon  plugs  and  pins  to  be  connected.  The  other  eight  tasks  use  the  HMD  and  an 
unenhanced  presentation  of  the  cannon  plugs  and  pins  to  be  connected. 


Performance  Measurement 


The  following  guidelines  are  used  by  the  experimenter  to  control  the  data 
collection: 

1 .  This  evaluation  is  concerned  with  cannon  plug  pin  selection/connection  error 
rates  and  task  completion  times  and  not  the  actual  operation  of  the  HMD. 

2.  If  a  technician  is  experiencing  difficulty  in  operating  the  HMD  or  any  of  its 
components  the  experimenter  helps  the  technician  in  the  proper  equipment 
operation  prior  to  beginning  the  task. 

3.  If  the  HMD  or  any  of  its  related  components  fail  during  the  test,  or  the 
technician  has  to  stop  the  test  for  any  reason  that  test  is  aborted  and  the 
technician’s  data  is  eliminated  from  the  analysis. 

4.  There  is  no  set  time  limit  for  task  completion. 

5.  Technicians  are  allowed  only  one  selection/connection  attempt  per  task. 


Conducting  the  Experiment 


Sequence  of  Events 


•  Completion  of  a  Human  Use  Research  Committee  (HURC)  Form. 

•  Completion  of  the  Personal  Background  Form. 

•  Introduction  to  the  experiment,  equipment,  and  test  method. 
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•  Hands-on  training  on  the  use  of  the  HMD. 

•  Practice  session  with  the  HMD  and  practice  cannon  plug. 

•  Technician  performs  the  sixteen  experiment  tasks. 

•  Experimenter  debriefs  technicians,  administers  the  questionnaire,  and 
conducts  the  structured  interview. 

Completion  of  Initial  Forms 

The  test  subjects  complete  a  HURC  Form,  Appendix  F.  This  form  outlines  their 
rights  and  the  test  administrators  responsibilities  as  applied  to  the  experiment.  The  test 
subjects  also  complete  the  Personal  Background  Form,  Appendix  E.  This  form  provides 
demographic  data  and  allows  one  more  check  to  ensure  that  the  test  subject  meets  the 
criteria  set  forth  for  inclusion  in  this  experiment.  The  test  subjects  are  then  assigned  one 
of  the  28  test  subject  numbers. 

Introduction 

The  technicians  receive  a  description  of  the  purpose  and  preliminary  instructions 
for  the  experiment.  Preliminary  instructions  include  the  responsibility  of  the  test 
participant,  test  administrator,  and  how  the  collected  data  is  to  be  used.  Technicians  are 
reminded  that  their  participation  is  on  a  voluntary  basis.  Data  collected  is  not  associated 
with  their  name,  only  with  an  assigned  test  subject  number.  Task  performance  and 
answers  to  the  User  Evaluation  Questionnaire  or  structured  interview  will  not  affect  their 
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job  performance  ratings  in  any  way.  Appendix  H  contains  the  instructions  that  the 
experimenter  will  read  to  the  technicians. 

Training 

Technicians  receive  initial  training  on  the  HMD  -  its  controls,  its  operation,  and 
its  proper  wear  (Appendix  I  contains  the  training  instructions). 

Practice 

Technicians  perform  four  practice  test  trials  using  the  rectangular  breakout  box 
assembly.  This  practice  is  intended  to  allow  the  test  subject  to  familiarize  themselves 
with  the  HMD  use  and  experimental  procedure.  During  the  training  sessions  the 
experimenter  is  available  to  answer  any  questions  that  arise.  After  the  technician  is 
comfortable  in  the  use  of  the  HMD  the  experiment  begins. 

Task  Completion 

The  test  subjects  perform  sixteen  experimental  tasks,  eight  tasks  each  using  the 
two  data  presentation  methods  (unenhanced  and  unenhanced)  in  a  block  randomized 
presentation  order.  The  test  administrator  steps  through  the  program  as  the  test  subject 
indicates  he  or  she  is  ready  to  continue. 
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Debriefing 


After  the  experiment  is  complete,  the  User  Evaluation  Questionnaire,  Appendix 
A,  is  completed  by  the  test  subject  and  a  structured  interview.  Appendix  B,  is  conducted. 
The  test  subjects  are  debriefed  and  reminded  not  to  discuss  the  context  of  the  experiment 
with  anyone  else  until  all  the  test  subjects  are  run  and  all  the  data  is  collected  (debrief 
instructions  are  in  Appendix  J).  Technicians  also  receive  information  on  how  their  data 
from  the  experiment  will  be  used. 
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Appendix  E:  Personal  Background  Form 


Test  Subject  Number: 


1.  Time  in  Service: _ 

2.  Paygrade: _ 

3.  Time  at  Barksdale  AFB: _ 

4.  Are  Your  Current  Primary  Duties  Performing  Maintenance  Tasks  on  the 
Flightline  /  Bench?  (i.e.  not  in  an  office,  support  section,  QA,  or  driving  a  truck) 

YES  /  NO 

5.  Current  Continuous  Time  on  the  Flightline  /  Bench: _ 

6.  Current  AFSC: _ 

7.  Is  Your  Eyesight  at  Least  20/20  Corrected?  YES  /  NO 

8.  If  You  Wear  Glasses,  Do  You  Wear  Bifocals  or  Trifocals?  YES  /  NO 

9.  Any  Reason  Why  You  Feel  You  Can  NOT  Participate  in  This  Experiment  at 
This  Time? 

YES  /  NO 
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Appendix  F:  Human  Use  Research  Committee  (HURC)  Form 

Test  Subject  Number: _ 

Informed  Consent  Document  for  Cannon  Plug  Pin  Selection  Performance  Evaluation 

Experiment 

1 .  I _ have  been  asked  to  volunteer  as  a  subject  in  the  project  named  above.  The 

purpose  of  the  study  is  to  evaluate  the  effect  that  information  presented  on  a  monocular  Head-Mounted 
Display  (HMD)  has  on  technician  performance  of  cannon  plug  pin  selection  and  connection  tasks. 

2.  In  this  study  I  will  be  asked  to  perform  caimon  plug  pin  selection  and  connection  tasks.  I  will  be  asked 
to  complete  a  questionnaire  regarding  the  tasks.  I  will  be  required  to  work  with  a  Head-Mounted 
Display  (HMD).  The  session  may  be  videotaped.  The  videotape  from  this  session  will  not  be 
presented  or  distributed  for  any  purpose  other  than  data  analysis.  Data  from  the  videotapes  will  be 
recorded  and  the  tapes  will  then  be  erased  immediately  after  completion  of  the  entire  study. 

3.  My  participation  will  not  involve  risks  greater  than  I  encounter  on  a  daily  basis.  Potential  hazards  of 
looking  at  a  computer  screen  are  possible  (i.e.  eye  strain,  headaches).  The  likelihood  of  these 
occurrences  is  slight. 

4.  My  participation  will  assist  the  Air  Force  Institute  of  Technology,  the  University  of  Dayton  Research 
Institute,  and  Armstrong  Laboratory  ensure  that  collaborative  technologies  meet  the  users  needs.  The 
ultimate  benefit  of  this  study  will  be  to  make  personnel  more  effective  and  make  their  jobs  easier. 

5 .  (i)  Records  of  my  participation  in  this  study  may  only  be  disclosed  according  to  federal  law, 

including  the  Federal  Privacy  Act,  5  U.S.C.  552a,  and  its  implementing  regulations. 

(ii)  I  understand  my  entitlements  to  medical  and  dental  care  and/or  compensation  in  the  event 
of  injury  are  governed  by  federal  laws  and  regulations,  and  that  if  1  desire  further 
information  I  may  contact  the  BAFB  legal  office  at  456-2561. 

(iii)  If  an  unanticipated  event  occurs  during  my  participation  in  this  study,  1  will  be  informed.  If  I  am 
not  competent  at  the  time  to  understand  the  nature  of  the  event,  such  information 

will  be  brought  to  the  attention  of  my  next  of  kin. 

(iv)  The  decision  to  participate  in  this  research  is  completely  voluntary  on  my  part.  No  one  has 
coerced  or  intimidated  me  into  participating  in  this  program.  I  am  participating  because  I  want  to. 
Captain  Webb,  (937)  258-2568  has  adequately  answered  any  and  all  questions  I  have  about  this 
study,  my  participation,  and  the  procedures  involved.  1  understand  that  Capt  Webb  will  be 
available  to  answer  any  questions  concerning  procedures  throughout  this  study.  I  understand  that 
if  significant  findings  develop  during  the  course  of  this  research  which  may  relate  to  my  decision 
to  continue  participation,  I  will  be  informed.  1  further  understand  that  I  may  withdraw  this 
consent  at  any  time  and  discontinue  further  participation  in  this  study  without  prejudice  to  my 


entitlements. 

Volunteer  Signature 

SSN 

Date 

Investigator  Signature 

Date 

Wimess: 

Date 
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Appendix  G:  Experimenter  Observation  Form 


Test  Subject  Number: 


Task  Number  1:  C  E 

Task  Number  2:  C  E 


Task  Number  3:  C  E 

Task  Number  4:  C  E 


Task  Number  5:  C  E 

Task  Number  6:  C  E 

Task  Number  7:  C  E 


Task  Number  8:  C  E 
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O'O 


Test  Subject  Number: 


Task  Number  9:  C  E_ 

Task  Number  10:  C  E 

Task  Number  11:  C  E 


Task  Number  12:  C  E 


©A  O 

6  P 


lo  o  6^ 

[oocud) 


Task  Number  13:  C  E 


Task  Number  14:  C  E 

Task  Number  15:  C  E 


Task  Number  16:  C  E 
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Appendix  H:  Briefing  Instructions 


Introduction 

Thank  you  for  volunteering  to  be  a  test  subject  in  the  evaluation  of  the  effect  of 
graphical  display  enhancements  on  cannon  plug  pin  selection/coimection  error  rates.  I 
am  Captain  Robert  Webb  and  I  am  from  the  Air  Force  Institute  of  Technology  at  Wright- 
Patterson  AFB,  Ohio.  Mr.  Allen  Revels,  from  the  University  of  Dayton  Research 
Institute,  (point  to  Allen)  will  be  conducting  this  experiment  with  me.  We  are  performing 
this  experiment  in  conjunction  with  Armstrong  Laboratory  and  the  University  of  Dayton 
Research  Institute,  both  also  at  Wright-Patterson.  This  research  will  also  fulfill  my 
masters  thesis  requirement. 

Purpose 

The  objective  of  this  research  is  to  study  the  effect  two  different  graphical  data 
presentation  methods  using  a  digital  Head-Mounted  Display  (Hold  up  the  HMD)  has  on 
cannon  plug  pin  identification  and  selection  tasks  (Hold  up  one  of  the  Cannon  Plugs). 

The  information  obtained  form  this  experiment  will  be  used  to  support  the 
continued  development  of  digitally  displayed  maintenance  data  with  the  ultimate  goal 
being  improved  maintenance  through  the  integration  of  humans  and  computer 
technology. 
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Experiment  Description 


Avionics  maintenance  technicians  are  participating  in  this  experiment.  All  of  the 
technicians  will  perform  tasks  using  both  presentation  methods.  All  of  the  technicians 
will  be  asked  to  fill  out  a  HURC  Form  (hold  up  a  copy  of  the  form).  This  form  spells  out 
your  rights  and  our  responsibilities  to  you  for  participating  in  this  experiment. 
Additionally,  each  of  you  will  fill  out  a  Personal  Background  Form  (hold  up  a  copy  of  the 
form).  This  form  documents  your  military  experience  and  background. 

The  information  collected  in  this  experiment  will  not  be  associated  with  your 
name,  only  a  test  subject  number.  The  data  collected  will  not  be  related  to  your  job 
performance,  nor  will  it  be  released  to  your  supervisors.  All  data  and  comments  collected 
will  be  completely  confidential. 

Each  technician  will  perform  sixteen  cannon  plug  pin  selection  tasks.  Technicians 
will  be  required  to  locate  and  connect  test  adapters  to  specific  pins  on  the  test  cannon 
plugs.  1  will  also  be  taking  observation  notes  while  you  perform  the  tasks.  Please  work 
as  quickly  and  accurately  as  possible. 

Your  test  session  may  be  videotaped.  If  it  is,  the  videotape  from  this  session  will 
not  be  presented  or  distributed  for  any  purpose  other  than  data  analysis.  The  videotape 
will  be  erased  upon  completion  of  the  final  report  of  this  experiment. 

After  you  complete  all  of  the  tasks  and  the  experimental  data  collection  is 
complete,  you  will  be  asked  to  complete  a  questionnaire  on  the  HMD  and  displayed 
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information.  I  will  then  talk  with  you  about  your  observations  and  impressions  of  the 
experiment  as  a  whole. 

The  test  sequence  of  events  will  be  as  follows; 

•  Completion  of  a  Human  Use  Research  Committee  (HURC)  Form  and 
Personal  Background  Form. 

•  Introduction  to  the  experiment,  equipment,  and  test  method. 

•  Hands-on  training  on  the  use  of  the  HMD. 

•  Practice  session  with  the  HMD  and  rectangular  cannon  plug. 

•  Technician  performs  the  sixteen  experiment  tasks. 

•  Completion  of  the  User  Evaluation  Questionnaire,  exit  interview,  and  debrief 
Do  you  have  any  questions  at  this  time? 
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Appendix  I:  Training  Instructions 


Using  the  HMD 

You  will  be  using  a  monocular  Head-Mounted  Display  (HMD)  as  your  source  of 
task  information  for  this  experiment.  I  am  going  to  provide  you  instructions  and  practice 
on  its  basic  operation  before  the  experiment  gets  underway.  Please  feel  free  to  ask  any 
questions  at  any  time  if  the  instructions  and  procedures  described  are  not  clear. 

(Show  the  technician  the  HMD  and  point  out  the  controls  as  they  are  discussed.) 
This  is  the  HMD.  It  is  a  back-lit  monochrome  LCD,  which  means  all  information 
displayed  is  in  shades  of  gray.  The  knob  directly  behind  the  screen  is  the  contrast  knob. 
The  knob  to  the  left  of  the  screen  on  the  head  band  is  the  screen  intensity  knob.  The  head 
band  can  be  adjusted  using  the  elastic  straps  on  the  head  band.  This  HMD  is  also 
equipped  with  voice  input  capabilities  which  we  will  not  be  using  for  this  experiment,  so 
the  ear  piece  and  microphone  on  the  head  band  are  of  no  importance.  Let’s  put  this  on 
and  get  you  use  to  wearing  it  (put  the  HMD  on  the  technician  and  allow  them  to  get  use 
to  wearing  it). 

The  information  that  will  be  presented  is  completely  menu  driven  (ensure  the 
HMD  is  on  and  the  program  introduction  page  is  showing).  You  will  only  need  to  follow 
the  instructions  on  the  screen  and  tell  us  when  you  are  ready  to  continue. 
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Cannon  Plug  Assemblies 


Five  different  cannon  plug  connector  assemblies  will  be  used  during  this 
experiment  (show  them  to  the  technician  and  tell  how  many  pins  are  on  each).  Note  that 
some  have  the  pins  numerically  referenced  and  some  are  alphabetically  referenced. 

Recall  that  on  alphabetically  referenced  cannon  plugs  the  letters  L,  O,  Q,  t,  and  <%  are  not 
used.  The  four  circular  cannon  plugs  will  be  used  in  the  experiment  itself  and  the 
rectangular  cannon  plug  is  used  for  orientation  and  practice  session. 

Test  Sequence 

The  test  program  is  a  menu  driven  presentation  (show  pictures  of  representative 
screens  as  the  following  sequence  is  discussed).  You  will  only  have  to  tell  us  when  you 
are  ready  to  proceed  and  we  will  advance  the  program.  The  test  sequence  will  start  when 
the  "'START'  button  is  clicked  and  will  conclude  when  the  "DONE"  button  is  clicked. 
(Ensure  that  the  technician  can  see  what  is  on  the  HMD  screen  and  understands  the 
instructions  being  presented.) 

The  screen  before  the  start  of  the  test  will  tell  you  what  connector  to  use.  We  will 
give  you  the  proper  connector  and  test  adapters.  Orient  yourself  with  the  caimon  plug  pin 
layout  and  test  adapters  before  telling  us  that  you  are  ready  to  continue.  The  next  screen 
will  tell  you  which  pins  to  coimect  and  provide  a  graphical  depiction  of  the  cannon  plug. 
Coimect  the  pins  and  tell  us  when  you  are  completed  with  the  task. 
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The  experiment  sequence  is: 

•  Four  practice  tasks  using  the  rectangular  cannon  plug. 

•  Sixteen  tasks  using  the  circular  cannon  plugs. 

•  Termination  of  the  test  sequence. 

After  completing  each  task  hand  the  cannon  plug  assembly,  with  the  test  adapters 
still  in  place,  to  me.  Again,  please  work  as  quickly  and  accurately  as  possible. 

This  concludes  the  instructions  and  practice  on  the  use  of  the  HMD  and  test 
equipment.  Do  you  have  any  questions  regarding  its  use  before  the  practice  session  gets 
underway? 
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Appendix  J:  Debriefing  Instructions 


Thank  you  for  participating  in  this  experiment.  The  purpose  of  this  experiment 
was  to  compare  cannon  plug  pin  selection  and  connection  performance  using  two 
graphical  presentation  methods  on  a  monocular  Head-Mounted  Display  (HMD).  The 
information  from  this  evaluation  will  be  used  in  future  research  in  support  of  the  selection 
of  a  display  technology  for  future  weapon  systems. 

None  of  the  information  received  or  data  collected  will  be  associated  with  your 
name.  Experiment  write-ups  wilt  describe  the  data  only  by  test  subject  number.  Do  you 
have  any  other  comments  or  questions  about  this  experiment? 

A  copy  of  the  final  thesis  will  be  sent  to  the  2d  Bomb  Wing  so  that  the 
participants  can  read  the  entire  research. 

Thanks  again  for  you  participation.  Please  do  not  discuss  any  aspect  of  this 
experiment  with  anyone  until  all  of  the  test  data  has  been  collected. 
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Appendix  K:  Test  Data 


The  following  is  the  raw  test  data  that  was  collected  at  Barksdale  AFB,  LA  in 
March  1 997  along  with  some  initial  data  analysis. 

Raw  Error  Data 


Table  16:  Raw  Error  Data 


Test 

Subject 

1 

2 

3 

4 

5 

6 

T  ria  1 

7  8 

Number 

9  1 0 

1  1 

1  2 

1  3 

1  4 

1  5 

1  6 

Total 

1 

1 

1 

1 

1 

1 

4 

1 

1 

2 

5 

1 

1 

1 

3 

6 

1 

1 

1 

1 

1 

1 

6 

7 

1 

1 

2 

9 

1 

1 

1 

1 

4 

1  0 

1 

1 

2 

1  1 

1 

1 

2 

1  3 

1 

1 

2 

1  4 

1 

1 

1 

1 

4 

1  5 

1 

1 

1 

3 

1  6 

1 

1 

1 

3 

1  7 

1 

1 

2 

1  8 

1 

1 

1 

3 

1  9 

1 

1 

1 

1 

4 

20 

1 

1 

1 

1 

1 

1 

1 

7 

21 

0 

22 

1 

1 

2 

23 

1 

1 

24 

1 

1 

26 

1 

1 

2 

27 

1 

1 

1 

1 

1 

1 

6 

28 

1 

1 

1 

3 

T  0  ta  I : 

3 

5 

6 

6 

2 

5 

2 

9 

4 

2 

6 

3 

6 

4 

3 

3 

69 

*  A  “  1  ”  indicates  an  error  occurred. 

*  Test  Subject  Number  1  ’s  data  was  eliminated  from  analysis  because  the  subject  was  given  the  wrong 
cannon  plug  for  trial  number  5.  This  caused  the  trial  time  data  to  be  bad  and  thus  the  entire  data  set  to  be 
unused  for  analysis. 

*  Test  Subject  Numbers  8  and  12  were  eliminated  from  analysis  because  both  test  subjects  indicated  that 
they  neither  saw  nor  used  the  enhanced  graphical  presentations  to  locate  the  cannon  plug  pins  during  the 
test.  This  invalidated  the  enhanced  comparison  of  their  data  and  made  it  unusable  for  analysis. 

*  Test  Subject  Number  25’s  data  was  eliminated  from  analysis  because  there  was  an  error  in  the  test 
program  sequencing.  The  test  subject  received  the  wrong  enhancements  and  thus  the  wrong  conditions 
were  presented. 
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Error  Data  Selection 


Correct  Pin 

Condition  Combinations 


S  - 


N  -s 
F  -X 
C  -  V 
S  -h 


24-46 
15  -  59 
6-54 
47-67 


24-46 

15-59 

6-54 

47-67 


Actually  Selected  Pin  Combinations 


5- 12 

3-11 

6- 7 

5- 12 

3-11 

6- 7 

5-12 

5  - 12 

1  -10 

1  -10 

5-12 

5-12 

5  -  12 

5  - 12 

N  -S 

N-r 

N  -  r 

N  -S 

P  -T 

N  -  u 

N  -  u 

N  -S 

P  -s 

N  -  AA 

N  -t 

E  -X 

F  -w 

F  -X 

F  -  w 

F  -GG 

C  -T 

C  -  V 

C  -X 

C  -  V 

C  -V 

C  -  V 

C  -  V 

C  -u 

C  -u 

S  -g 

H  -S 

S  -g 

M  -s 

L  -  s 

F-y 

F  -X 

C  -  u 

C  -  w 

S  -i 

24-43 

23-36 

25-46 

15-63 

25-59 

13-56 

16-59 

26-70 

6-70 

6-65 

4-6 

6-67 

37-57 

48-51 

47-51 

15-39 

6-52 
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Raw  Time  Data 


Table  18:  Raw  Time  Data 


Subject 

Number 

Trial 

Number 

Enhanced 

Test 

Set 

Elapsed 

Time 

Suited 

Number 

Trial 

Number 

Enhanced 

T^ 

Set 

Bapsed 

Time 

2 

1 

10 

16 

5 

1 

FALSE 

4 

11 

2 

2 

11 

9 

5 

TRUE 

6 

2 

3 

16 

16 

5 

FALSE 

17 

2 

4 

FALSE 

1 

55 

5 

TRUE 

2 

5 

TRUE 

20 

5 

TRUE 

WM 

2 

6 

FALSE 

54 

5 

FALSE 

■1 

10 

2 

7 

FALSE 

13 

5 

FALSE 

42 

2 

8 

FALSE 

13 

5 

8 

TRUE 

13 

9 

2 

9 

TRUE 

33 

5 

9 

FALSE 

14 

11 

2 

10 

TRUE 

13 

5 

10 

TRUE 

12 

6 

2 

11 

FALSE 

14 

44 

5 

11 

TRUE 

9 

11 

2 

12 

TRUE 

16 

5 

12 

FALSE 

15 

16 

2 

13 

FALSE 

10 

5 

13 

FALSE 

5 

15 

2 

14 

TRUE 

30 

5 

14 

TRUE 

3 

7 

2 

15 

FALSE 

26 

5 

15 

FALSE 

16 

7 

2 

16 

FALSE 

12 

18 

5 

16 

TRUE 

10 

6 

3 

1 

FALSE 

10 

32 

6 

1 

11 

11 

3 

2 

TRUE 

13 

32 

6 

2 

FALSE 

8 

16 

3 

3 

FALSE 

15 

13 

6 

3 

FALSE 

5 

101 

3 

4 

TRUE 

16 

12 

6 

4 

TRUE 

10 

13 

3 

5 

TRUE 

7 

11 

6 

5 

TRUE 

3 

7 

3 

6 

FALSE 

5 

70 

6 

6 

TRUE 

4 

10 

3 

7 

FALSE 

12 

20 

6 

7 

TRUE 

1 

29 

3 

8 

TRUE 

14 

18 

6 

8 

FALSE 

6 

92 

3 

9 

FALSE 

3 

12 

6 

9 

TRUE 

14 

7 

3 

10 

TRUE 

8 

8 

6 

10 

TRUE 

16 

8 

3 

11 

FALSE 

1 

48 

6 

11 

TRUE 

13 

13 

3 

12 

FALSE 

6 

30 

6 

12 

FALSE 

15 

10 

3 

13 

TRUE 

2 

16 

6 

13 

FALSE 

9 

18 

3 

14 

TRUE 

11 

8 

6 

14 

TRUE 

7 

9 

3 

15 

FALSE 

4 

14 

6 

15 

FALSE 

12 

11 

3 

16 

TRUE 

9 

23 

6 

16 

FALSE 

2 

46 

4 

1 

FALSE 

5 

96 

7 

1 

TRUE 

2 

23 

4 

2 

FALSE 

11 

13 

7 

2 

TRUE 

4 

17 

4 

3 

FALSE 

8 

11 

7 

3 

TRUE 

7 

10 

4 

4 

TRUE 

10 

63 

7 

4 

TRUE 

9 

11 

4 

5 

TRUE 

7 

13 

7 

5 

FALSE 

6 

41 

4 

6 

TRUE 

12 

14 

7 

6 

FALSE 

12 

12 

4 

7 

TRUE 

9 

17 

7 

7 

FALSE 

3 

15 

4 

8 

FALSE 

14 

22 

7 

8 

FALSE 

13 

44 

4 

9 

TRUE 

6 

31 

7 

9 

TRUE 

5 

15 

4 

10 

FALSE 

13 

44 

7 

10 

FALSE 

11 

6 

4 

11 

FALSE 

15 

19 

7 

11 

TRUE 

16 

11 

4 

12 

TRUE 

16 

16 

7 

12 

TRUE 

10 

12 

4 

13 

FALSE 

2 

18 

7 

13 

FALSE 

1 

17 

4 

14 

TRUE 

1 

25 

7 

14 

FALSE 

8 

11 

4 

15 

TRUE 

3 

12 

7 

15 

FALSE 

14 

36 

4 

16 

FALSE 

4 

15 

7 

16 

TRUE 

15 

21 
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Table  18  Continued 


Subject 

Trial 

Test 

Elapsed 

Number 

Number 

Enhanced 

Set 

Time 

9 

1 

TRUE 

8 

9 

g 

2 

TRUE 

7 

7 

9 

3 

TRUE 

5 

10 

9 

4 

FALSE 

10 

20 

9 

5 

FALSE 

15 

8 

9 

6 

FALSE 

1 

26 

9 

7 

TRUE 

2 

17 

9 

8 

FALSE 

16 

11 

9 

9 

FALSE 

4 

8 

9 

10 

TRUE 

13 

17 

9 

11 

TRUE 

3 

7 

9 

12 

FALSE 

14 

11 

9 

13 

FALSE 

11 

10 

9 

14 

FALSE 

9 

31 

9 

15 

TRUE 

6 

10 

9 

16 

TRUE 

12 

10 

10 

1 

FALSE 

2 

47 

10 

2 

TRUE 

4 

24 

10 

3 

FALSE 

7 

26 

10 

4 

TRUE 

13 

47 

10 

5 

FALSE 

12 

36 

10 

6 

FALSE 

1 

61 

10 

7 

TRUE 

6 

38 

10 

8 

TRUE 

3 

22 

10 

9 

TRUE 

11 

11 

10 

10 

FALSE 

10 

93 

10 

11 

FALSE 

16 

20 

10 

12 

FALSE 

5 

110 

10 

13 

FALSE 

15 

21 

10 

14 

TRUE 

8 

12 

10 

15 

TRUE 

9 

27 

10 

16 

TRUE 

14 

21 

11 

1 

TRUE 

12 

12 

11 

2 

TRUE 

5 

32 

11 

3 

FALSE 

15 

12 

11 

4 

FALSE 

10 

66 

11 

5 

FALSE 

16 

38 

11 

6 

TRUE 

7 

10 

11 

7 

FALSE 

13 

65 

11 

8 

TRUE 

6 

64 

11 

9 

TRUE 

14 

24 

11 

10 

FALSE 

8 

11 

11 

11 

TRUE 

11 

12 

11 

12 

TRUE 

1 

31 

11 

13 

FALSE 

2 

75 

11 

14 

FALSE 

9 

41 

11 

15 

FALSE 

3 

9 

11 

16 

TRUE 

4 

21 

Sut^ect 

Trial 

Test 

Elapsed 

Number 

Number 

Enhanced 

Set 

Time 

13 

1 

FALSE 

14 

36 

13 

2 

FALSE 

4 

13 

13 

3 

TRUE 

13 

37 

13 

4 

TRUE 

7 

10 

13 

5 

TRUE 

10 

9 

13 

6 

FALSE 

15 

15 

13 

7 

TRUE 

16 

10 

13 

8 

FALSE 

9 

29 

13 

9 

FALSE 

6 

63 

13 

10 

TRUE 

5 

18 

13 

11 

TRUE 

8 

10 

13 

12 

FALSE 

11 

11 

13 

13 

TRUE 

2 

14 

13 

14 

TRUE 

3 

6 

13 

15 

FALSE 

12 

17 

13 

16 

FALSE 

1 

22 

14 

1 

FALSE 

8 

14 

14 

2 

TRUE 

5 

33 

14 

3 

FALSE 

10 

21 

14 

4 

TRUE 

11 

15 

14 

5 

TRUE 

4 

12 

14 

6 

FALSE 

13 

44 

14 

7 

TRUE 

14 

11 

14 

8 

FALSE 

3 

15 

14 

9 

TRUE 

9 

17 

14 

10 

FALSE 

15 

14 

14 

11 

FALSE 

2 

17 

14 

12 

FALSE 

16 

13 

14 

13 

TRUE 

12 

8 

14 

14 

TRUE 

6 

8 

14 

15 

FALSE 

1 

47 

14 

16 

TRUE 

7 

9 

15 

1 

FALSE 

12 

15 

15 

2 

FALSE 

15 

13 

15 

3 

FALSE 

9 

13 

15 

4 

FALSE 

2 

27 

15 

5 

TRUE 

4 

11 

15 

6 

TRUE 

13 

33 

15 

7 

TRUE 

3 

8 

15 

8 

TRUE 

6 

21 

15 

9 

TRUE 

16 

9 

15 

10 

TRUE 

11 

8 

15 

11 

TRUE 

5 

14 

15 

12 

TRUE 

14 

10 

15 

13 

FALSE 

8 

19 

15 

14 

FALSE 

7 

8 

15 

15 

FALSE 

1 

22 

15 

16 

FALSE 

10 

21 
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Table  18  Continued 


Subject 

Number 

Trial 

Number 

Enhanced 

Test 

Set 

Elapsed 

Time 

Subject 

Number 

Trial 

Number 

Enhanced 

Test 

Set 

Elapsed 

Time 

16 

1 

FALSE 

10 

12 

19 

TRUE 

13 

38 

16 

FALSE 

11 

10 

19 

FALSE 

12 

13 

16 

FALSE 

12 

19 

19 

FALSE 

11 

11 

16 

FALSE 

17 

19 

TRUE 

14 

16 

16 

5 

TRUE 

19 

TRUE 

4 

9 

16 

6 

TRUE 

19 

FALSE 

5 

31 

16 

TRUE 

17 

19 

TRUE 

15 

10 

16 

TRUE 

16 

19 

FALSE 

2 

87 

16 

TRUE 

14 

18 

19 

9 

FALSE 

3 

22 

16 

10 

FALSE 

8 

12 

19 

10 

TRUE 

16 

11 

16 

11 

TRUE 

13 

18 

19 

11 

FALSE 

1 

16 

16 

12 

FALSE 

15 

20 

19 

12 

FALSE 

6 

37 

16 

13 

TRUE 

16 

11 

19 

13 

FALSE 

8 

11 

16 

14 

TRUE 

11 

19 

14 

TRUE 

9 

22 

16 

15 

FALSE 

38 

19 

15 

TRUE 

7 

7 

16 

16 

FALSE 

16 

19 

16 

TRUE 

10 

8 

17 

1 

12 

14 

20 

1 

TRUE 

4 

12 

17 

2 

FALSE 

6 

53 

20 

2 

TRUE 

14 

11 

17 

3 

TRUE 

5 

64 

20 

3 

TRUE 

13 

29 

17 

4 

FALSE 

3 

7 

20 

4 

FALSE 

15 

13 

17 

5 

TRUE 

2 

51 

20 

5 

TRUE 

3 

9 

17 

6 

TRUE 

15 

7 

20 

6 

FALSE 

8 

12 

17 

7 

FALSE 

9 

26 

20 

7 

FALSE 

9 

20 

17 

8 

TRUE 

8 

14 

20 

8 

FALSE 

2 

44 

17 

9 

TRUE 

10 

8 

20 

9 

FALSE 

6 

9 

17 

10 

FALSE 

16 

12 

20 

10 

TRUE 

5 

25 

17 

11 

FALSE 

13 

26 

20 

11 

TRUE 

16 

17 

17 

12 

FALSE 

11 

7 

20 

12 

TRUE 

11 

10 

17 

13 

TRUE 

1 

18 

20 

13 

FALSE 

7 

7 

17 

14 

FALSE 

14 

31 

20 

14 

FALSE 

12 

9 

17 

15 

TRUE 

7 

9 

20 

15 

TRUE 

10 

12 

17 

16 

TRUE 

4 

17 

20 

16 

FALSE 

1 

59 

18 

1 

FALSE 

9 

30 

21 

1 

TRUE 

12 

19 

18 

2 

TRUE 

2 

14 

21 

2 

TRUE 

15 

7 

18 

3 

TRUE 

7 

15 

21 

3 

FALSE 

1 

25 

18 

4 

TRUE 

4 

19 

21 

4 

FALSE 

2 

21 

18 

5 

FALSE 

12 

9 

21 

5 

TRUE 

13 

18 

18 

6 

FALSE 

3 

7 

21 

6 

FALSE 

7 

12 

18 

7 

TRUE 

1 

25 

21 

7 

FALSE 

4 

12 

18 

8 

FALSE 

6 

25 

21 

8 

TRUE 

6 

11 

18 

9 

FALSE 

8 

10 

21 

9 

TRUE 

14 

7 

18 

10 

FALSE 

15 

14 

21 

10 

TRUE 

3 

7 

18 

11 

FALSE 

5 

20 

21 

11 

FALSE 

16 

8 

18 

12 

FALSE 

10 

14 

21 

12 

FALSE 

5 

13 

18 

13 

TRUE 

16 

7 

21 

13 

FALSE 

10 

33 

18 

14 

TRUE 

11 

6 

21 

14 

TRUE 

8 

8 

18 

15 

TRUE 

13 

20 

21 

15 

TRUE 

9 

13 

18 

16 

TRUE 

14 

6 

21 

16 

FALSE 

11 

9 
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Table  18  Continued 


Subject 

Number 

Trial 

Number 

Enhanced 

Test 

Set 

Elapsed 

Time 

Subject 

Number 

Trial 

Number 

Enhanced 

Test 

Set 

Elapsed 

Time 

22 

1 

TRUE 

5 

41 

26 

1 

TRUE 

8 

15 

22 

2 

FALSE 

8 

28 

26 

2 

TRUE 

15 

11 

22 

3 

FALSE 

11 

13 

26 

3 

FALSE 

13 

39 

22 

4 

TRUE 

6 

24 

26 

4 

TRUE 

10 

33 

22 

5 

TRUE 

3 

8 

26 

5 

FALSE 

16 

12 

22 

6 

TRUE 

12 

17 

26 

6 

FALSE 

3 

11 

22 

7 

FALSE 

13 

39 

26 

7 

FALSE 

2 

44 

22 

8 

FALSE 

2 

33 

26 

8 

TRUE 

9 

23 

22 

9 

FALSE 

9 

24 

26 

9 

TRUE 

12 

10 

22 

10 

FALSE 

7 

10 

26 

10 

TRUE 

11 

10 

22 

11 

TRUE 

10 

13 

26 

11 

TRUE 

5 

11 

22 

12 

FALSE 

4 

13 

26 

12 

FALSE 

14 

30 

22 

13 

TRUE 

16 

15 

26 

13 

TRUE 

6 

18 

22 

14 

FALSE 

14 

26 

26 

14 

FALSE 

7 

14 

22 

15 

TRUE 

1 

21 

26 

15 

FALSE 

1 

19 

22 

16 

TRUE 

15 

7 

26 

16 

FALSE 

4 

8 

23 

1 

FALSE 

13 

41 

27 

1 

FALSE 

16 

15 

23 

2 

TRUE 

10 

87 

27 

2 

TRUE 

10 

34 

23 

3 

TRUE 

8 

21 

27 

3 

FALSE 

15 

11 

23 

4 

TRUE 

15 

13 

27 

4 

TRUE 

5 

40 

23 

5 

FALSE 

6 

60 

27 

5 

TRUE 

3 

9 

23 

6 

FALSE 

4 

16 

27 

6 

FALSE 

6 

39 

23 

7 

TRUE 

5 

23 

27 

7 

TRUE 

8 

17 

23 

8 

FALSE 

11 

9 

27 

8 

FALSE 

13 

76 

23 

9 

FALSE 

9 

81 

27 

9 

TRUE 

9 

50 

23 

10 

FALSE 

7 

8 

27 

10 

FALSE 

7 

8 

23 

11 

FALSE 

2 

54 

27 

11 

TRUE 

2 

34 

23 

12 

FALSE 

12 

11 

27 

12 

FALSE 

12 

18 

23 

13 

TRUE 

14 

29 

27 

13 

FALSE 

1 

35 

23 

14 

TRUE 

3 

10 

27 

14 

TRUE 

4 

12 

23 

15 

TRUE 

1 

13 

27 

15 

FALSE 

14 

42 

23 

16 

TRUE 

16 

9 

27 

16 

TRUE 

11 

24 

24 

1 

FALSE 

16 

14 

28 

1 

FALSE 

4 

16 

24 

2 

TRUE 

2 

26 

28 

2 

TRUE 

3 

10 

24 

3 

FALSE 

3 

10 

28 

3 

TRUE 

13 

35 

24 

4 

TRUE 

13 

38 

28 

4 

FALSE 

10 

76 

24 

5 

FALSE 

9 

43 

28 

5 

TRUE 

2 

10 

24 

6 

TRUE 

12 

10 

28 

6 

FALSE 

9 

48 

24 

7 

TRUE 

11 

9 

28 

7 

TRUE 

12 

12 

24 

8 

FALSE 

10 

91 

28 

8 

FALSE 

15 

21 

24 

9 

TRUE 

4 

7 

28 

9 

TRUE 

11 

5 

24 

10 

FALSE 

1 

24 

28 

10 

TRUE 

1 

21 

24 

11 

FALSE 

7 

14 

28 

11 

TRUE 

16 

9 

24 

12 

TRUE 

14 

10 

28 

12 

FALSE 

14 

14 

24 

13 

FALSE 

6 

71 

28 

13 

FALSE 

8 

9 

24 

14 

TRUE 

15 

8 

28 

14 

TRUE 

6 

9 

24 

15 

FALSE 

8 

11 

28 

15 

FALSE 

7 

8 

24 

16 

TRUE 

5 

43 

28 

16 

FALSE 

5 

17 
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Time  Output  by  Test  Subject  and  Trial 


Table  19:  Time  Output  by  Test  Subject  and  Trial 


Test 

Subject 

1 

2 

3 

4 

5 

6 

Trial  Number 

7  8  9  10 

11 

12 

13 

14 

15 

16 

2 

16 

9 

16 

55 

20 

54 

13 

13 

33 

13 

44 

16 

10 

30 

26 

18 

3 

32 

32 

13 

12 

11 

70 

20 

18 

12 

8 

48 

30 

16 

8 

14 

23 

4 

96 

13 

11 

63 

13 

14 

17 

22 

31 

44 

19 

16 

18 

25 

12 

15 

5 

11 

6 

17 

8 

8 

10 

42 

9 

11 

6 

11 

16 

15 

7 

7 

6 

6 

11 

16 

101 

13 

7 

10 

29 

92 

7 

8 

13 

10 

18 

9 

11 

46 

7 

23 

17 

10 

11 

41 

12 

15 

44 

15 

6 

11 

12 

17 

11 

36 

21 

9 

9 

7 

10 

20 

8 

26 

17 

11 

8 

17 

7 

11 

10 

31 

10 

10 

10 

47 

24 

26 

47 

36 

61 

38 

22 

11 

93 

20 

110 

21 

12 

27 

21 

11 

12 

32 

12 

66 

38 

10 

65 

64 

24 

11 

12 

31 

75 

41 

9 

21 

13 

36 

13 

37 

10 

9 

15 

10 

29 

63 

18 

10 

11 

14 

6 

17 

22 

14 

14 

33 

21 

15 

12 

44 

11 

15 

17 

14 

17 

13 

8 

8 

47 

9 

15 

15 

13 

13 

27 

11 

33 

8 

21 

9 

8 

14 

10 

19 

8 

22 

21 

16 

12 

10 

19 

17 

8 

7 

17 

16 

18 

12 

18 

20 

11 

11 

38 

16 

17 

14 

53 

64 

7 

51 

7 

26 

14 

8 

12 

26 

7 

18 

31 

9 

17 

18 

30 

14 

15 

19 

9 

7 

25 

25 

10 

14 

20 

14 

7 

6 

20 

6 

19 

38 

13 

11 

16 

9 

31 

10 

87 

22 

11 

16 

37 

11 

22 

7 

8 

20 

12 

11 

29 

13 

9 

12 

20 

44 

9 

25 

17 

10 

7 

9 

12 

59 

21 

19 

7 

25 

21 

18 

12 

12 

11 

7 

7 

8 

13 

33 

8 

13 

9 

22 

41 

28 

13 

24 

8 

17 

39 

33 

24 

10 

13 

13 

15 

26 

21 

7 

23 

41 

87 

21 

13 

60 

16 

23 

9 

81 

8 

54 

11 

29 

10 

13 

9 

24 

14 

26 

10 

38 

43 

10 

9 

91 

7 

24 

14 

10 

71 

8 

11 

43 

26 

15 

11 

39 

33 

12 

11 

44 

23 

10 

10 

11 

30 

18 

14 

19 

8 

27 

15 

34 

11 

40 

9 

39 

17 

76 

50 

8 

34 

18 

35 

12 

42 

24 

28 

16 

10 

35 

76 

10 

48 

12 

21 

5 

21 

9 

14 

9 

9 

8 

17 

*  The  Trial  Number  time  is  in  seconds.  The  Test  subject’s  trial  times  were  rounded  to 
the  nearest  second  for  analysis  purposes 
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Time  Data  by  Test  Condition 


Table  20:  Time  Data  by  Test  Condition 


Pin 

Test  Subject  Task  Time 

[sec) 

Condition 

Combinations 

2 

3 

4 

5 

6 

7 

9 

10 

11 

13 

14 

15 

WBBM 

13 

14 

15 

11 

8 

13 

11 

16 

11 

11 

14 

19 

■Ifl 

18 

20 

11 

12 

36 

17 

15 

7 

11 

20 

38 

13 

E2a 

A-F 

10 

17 

24 

21 

12 

11 

E2b 

D-J 

13 

8 

8 

9 

12 

10 

E2c 

E-K 

14 

6 

10 

12 

8 

E2d 

D-H 

16 

12 

16 

8 

11 

10 

9 

U3a 

9-10 

10 

12 

15 

9 

15 

U3b 

4-12 

10 

26 

8 

U3c 

2-11 

13 

11 

6 

10 

11 

U3d 

5-6 

13 

13 

19 

16 

10 

8 

21 

12 

15 

14 

13 

E3a 

9-10 

12 

7 

7 

7 

22 

6 

8 

E3b 

4-12 

16 

11 

13 

9 

10 

7 

10 

10 

9 

E3c 

2-11 

9 

8 

6 

11 

12 

15 

8 

E3d 

5-6 

21 

U5a 

N  -s 

18 

42 

46 

47 

75 

17 

27 

U5b 

F-x 

54 

30 

92 

41 

63 

U5c 

C-v 

32 

20 

93 

66 

21 

21 

U5d 

S-h 

44 

22 

11 

36 

11 

36 

E5a 

N  -s 

30 

16 

23 

17 

14 

E5b 

F-x 

31 

8 

10 

38 

64 

8 

21 

E5c 

C-v 

16 

63 

6 

13 

12 

9 

E5d 

S-h 

18 

7 

21 

24 

11 

10 

U9a 

24-46 

55 

48 

17 

17 

26 

61 

22 

47 

22 

U9b 

15-59 

26 

70 

96 

15 

101 

110 

U9c 

6-54 

18 

31 

41 

29 

13 

U9d 

47-67 

44 

44 

65 

44 

■E9a 

25 

~7S~ 

31 

E9b 

15-59 

15 

10 

32 

18 

33 

14 

E9c 

6-54 

20 

23 

17 

11 

11 

27 

17 

E9d 

47-67 

33 

32 

9 

13 

17 

47 

37 

33 
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Table  20  Continued 


Condition 

Pin 

Combinations 

16 

17 

Test  Subject  Task  Time  (sec) 
18  19  20  21  22  23  24 

26 

27 

28 

U2a 

A-F 

12 

13 

16 

8 

16 

U2b 

D  -  J 

12 

10 

11 

12 

28 

11 

9 

U2c 

E-K 

19 

14 

9 

13 

9 

11 

18 

U2d 

D-H 

12 

8 

14 

12 

15 

E2a 

8 

17 

19 

9 

12 

7 

12 

E2b 

14 

8 

21 

15 

17 

E2c 

E-K 

19 

17 

10 

10 

12 

E2d 

_  D-H _ 

11 

7 

11 

17 

15 

9 

9 

U3a 

7 

7 

22 

10 

11 

U3b 

7 

12 

10 

8 

14 

14 

8 

8 

U3c 

10 

7 

11 

9 

13 

9 

U3d 

20 

14 

13 

11 

21 

E3a 

9-10 

7 

9 

7 

8 

10 

9 

10 

E3b 

4-12 

11 

9 

15 

7 

E3c 

2  - 11 

6 

10 

9 

10 

24 

5 

E3d 

5-6 

7 

10 

7 

7 

13 

8 

11 

U5a 

N  -  s 

38 

87 

44 

21 

33 

54 

44 

U5b 

F-x 

53 

25 

37 

9 

60 

71 

39 

U5c 

C-v 

12 

14 

33 

91 

76 

U5d 

S  -h 

31 

26 

30 

42 

14 

E5a 

N  -s 

51 

14 

26 

34 

10 

E5b 

F-x 

16 

11 

24 

18 

9 

E5c 

C-v 

8 

8 

12 

13 

87 

33 

34 

E5d 

S-h 

18 

6 

16 

11 

7 

29 

10 

U9a 

24-46 

16 

59 

25 

24 

19 

35 

U9b 

15-59 

17 

20 

31 

13 

17 

U9c 

6-54 

16 

26 

30 

20 

24 

81 

43 

48 

U9d 

47-67 

26 

39 

41 

39 

76 

'  E9a 

~T7“ 

tim 

~ZS~ 

21 

13 

21 

E9b 

15-59 

64 

25 

41 

23 

43 

11 

40 

E9c 

6-54 

22 

13 

23 

50 

E9d 

47-67 

18 

20 

38 

29 

18 

38 

35 
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